- developers-list - monetdb.org

Announcement: New Mar2025 Feature release of MonetDB suite
by Sjoerd Mullender 27 Mar '25

27 Mar '25

The MonetDB Foundation is pleased to announce the Mar2025 feature release of the MonetDB suite of programs. More information about MonetDB can be found on our website at <https://www.monetdb.org/>. For details on this release, please see the release notes at <https://www.monetdb.org/release-notes/mar2025/>. As usual, the download location is <https://www.monetdb.org/downloads/>. The Mar2025 documentation can be found [here](https://www.monetdb.org/documentation-Mar2025/). ## Mar2025 Feature Release (11.53.3) ### - There is a new shared library called libmutils that contains some utility functions that are used by several programs. - Hot snapshot: allow member files larger than 64 GiB. By member files we mean the files inside the resulting .tar file, not the tar file itself. Huge member files are written using a GNU tar extension to the original tar format, which doesn't support more than 8 GiB. ### Client Package - Support for dumping databases from servers from before Jul2021 (11.41.X) has been removed. ### MonetDB Common - The implementation for the imprints index on numeric columns has been removed. It hasn't been used in years, and when it is enabled, it doesn't really make queries go faster. ### Geom Module - Removed type geometryA (geometry array). It was deprecated in the Jun2023 release (11.47.X) because there was no use for the type. ### MonetDB5 Server - Removed function bat.attach since it wasn't used. - Removed the MAL type "identifier" and supporting functions. There has never been an SQL interface to this type. - Removed the MAL type "color" and supporting functions. There has never been an SQL interface to this type. ### SQL Frontend - Added scalar functions: dayname(d date) and monthname(d date) returns varchar(10). - ranking window functions are now optimized into topn's For the grouped case we added the missing grouped/heap based topn implementation. - Added support for reading external data in a generic way via table returning function: proto_loader(string uri). The uri string value must start with the scheme name, ending with : character. Supported schemes are: monetdb: and odbc:. The monetdb scheme allows you to connect to a remote MonetDB server and retrieve the data of a specific table or view in a specific schema. The uri syntax: monetdb://[<host>[:<port>]]/<database>/<schema>/<table> Example: SELECT * FROM proto_loader('monetdb://127.0.0.1:50000/demo_db/sys/tables'); The odbc scheme allows you to connect to any ODBC data source via an ODBC driver and retrieve the data of a supplied query. The uri syntax: odbc:{{DSN|FILEDSN}=<data source name>|DRIVER=<path_to_driver>}; [<ODBC connection parameters>;]QUERY=<SQL query> For ODBC you normally configure a data source first. This is done using the ODBC administrator (on windows: odbcad32.exe, on linux: odbcinst). Once a data source for a specific ODBC driver has been setup using a unique name, you can reference it as: DSN=my_bigdata; or FILE_DSN=/home/usernm/dsns/my_bigdata.dsn; If you do not want to setup a data source, you can use DRIVER=...; to specify the ODBC driver program to use. However this also means you have to specify all the required connection parameters yourself, such as UID=...;PWD=...;DATABASE=...; etc. The QUERY=<SQL query> part is mandatory and must be specified at the end of the uri string, after the optional ODBC connection parameters. Examples: SELECT * FROM proto_loader( 'odbc:DSN=Postgres;UID=claude;PWD=monet;QUERY=SELECT * FROM customers'); SELECT * FROM proto_loader('odbc:DRIVER=/usr/lib64/libsqlite3odbc.so; Database=/home/martin/sqlite3/chinook.db;QUERY=SELECT * FROM customers'); Note that the 'odbc:' scheme is experimental and not enabled by default. To enable it, the MonetDB server has to be started with argument: --loadmodule odbc_loader - Extended the constant aggregate optimizer in order to eliminate aggregates with constant arguments whenever possible. - REMOTE TABLES and REPLICA TABLES now fully support the monetdb:// and monetdbs:// URL's introduced in Aug2024. Any mapi:monetdb:// URL's are normalized to the new style. - Add function sa_msettings_create() to allocate an msettings object using the arena allocator. - Unused helper function mapiuri_database() has been removed from rel_remote.h. - Added support for aggregates which order within the group such as quantile and which potentially order within the group such as group_concat. The ordering for such operators in now handled once in the relational plan. For this the create function statements can now have an optional order specification, using the keywords 'ORDERED' and 'WITH ORDER'. - Added support for recursive CTE's. - The SQL parser was cleaned up. This resulted in some keywords being used more strictly. If any of these keywords are to be used as column names, they have to be quoted using double quotes: AS, TABLE, COLUMN, DISTINCT, EXEC, EXECUTE. - Introduce the RETURNING clause for INSERT, UPDATE and DELETE statements. Specifying a RETURNING clause causes the SQL statement to return the modified records which can be queried using SELECT like expressions in the RETURNING clause. Aggregate functions are allowed. This is a common non-standard SQL extension. Examples: INSERT INTO foo values (1,10), (-1,-10) RETURNING i+2*j AS bar ---- 21 -21 UPDATE foo SET i = -i WHERE i >0 RETURNING sum(j), count(j) ---- -60|3 - Introduce division_min_scale SQL environment variable for specifying minimum scale of the division result. The default value is 3. ### Bug Fixes - [7101](https://github.com/MonetDB/MonetDB/issues/7101): Feature request: nextafter() in SQL - [7159](https://github.com/MonetDB/MonetDB/issues/7159): CREATE LOCAL TEMPORARY VIEW - [7331](https://github.com/MonetDB/MonetDB/issues/7331): Support RETURNING clause - [7578](https://github.com/MonetDB/MonetDB/issues/7578): explain result in Mal is truncated in large UDFs and their input bats is not shown - [7609](https://github.com/MonetDB/MonetDB/issues/7609): Upgrade 11.49.11 to 11.51.7 issues - [7611](https://github.com/MonetDB/MonetDB/issues/7611): Not possible to create table with multiple composite UNIQUE NULLS NOT DISTINCT constraints - [7614](https://github.com/MonetDB/MonetDB/issues/7614): Filter function creates a cartesian product when used with a view - [7615](https://github.com/MonetDB/MonetDB/issues/7615): Filter function creates a cartesian product when used with a view (2) - [7616](https://github.com/MonetDB/MonetDB/issues/7616): Filter function disappears - [7618](https://github.com/MonetDB/MonetDB/issues/7618): Tables loose their columns - [7619](https://github.com/MonetDB/MonetDB/issues/7619): Resource leak in prepared statements - [7621](https://github.com/MonetDB/MonetDB/issues/7621): crash on aggregate with case statement - [7622](https://github.com/MonetDB/MonetDB/issues/7622): In PREPARE queries with many parameters, information about parameters is truncated when sent to client. - [7623](https://github.com/MonetDB/MonetDB/issues/7623): Database crashed when using UPDATE xxx SET xxx RETURNING xx

1 0

Announcement: New Aug2024-SP2 Bugfix release of MonetDB suite
by Sjoerd Mullender 20 Dec '24

20 Dec '24

The MonetDB Foundation is pleased to announce the Aug2024-SP2 bugfix release of the MonetDB suite of programs. More information about MonetDB can be found on our website at <https://www.monetdb.org/>. For details on this release, please see the release notes at <https://www.monetdb.org/release-notes/aug2024/>. As usual, the download location is <https://www.monetdb.org/downloads/>. ### Bug Fixes - [7112](https://github.com/MonetDB/MonetDB/issues/7112): Need keyboard shortcut to interrupt query execution rather than session - [7205](https://github.com/MonetDB/MonetDB/issues/7205): Unpredictable performance when performing joins over nested queries - [7574](https://github.com/MonetDB/MonetDB/issues/7574): Assertion failure at `rel2bin_select` when using `STARTSWITH` - [7588](https://github.com/MonetDB/MonetDB/issues/7588): incorrect output with single row inputs for var_samp(c) over() - [7589](https://github.com/MonetDB/MonetDB/issues/7589): "SELECT * FROM sessions" crashes monetdb/e - [7593](https://github.com/MonetDB/MonetDB/issues/7593): A value is being returned with unnecessary scientific notation - [7595](https://github.com/MonetDB/MonetDB/issues/7595): SQLTestCase leaks pymonetdb connections - [7597](https://github.com/MonetDB/MonetDB/issues/7597): Upgrade + quick restart causes database inconsistency - [7599](https://github.com/MonetDB/MonetDB/issues/7599): str_to_date fails when combined with SQL CASE clause - [7602](https://github.com/MonetDB/MonetDB/issues/7602): COPY INTO from multiple files causes an assertion error. - [7603](https://github.com/MonetDB/MonetDB/issues/7603): COPY INTO from three or more files crashes the server. - [7604](https://github.com/MonetDB/MonetDB/issues/7604): file_loader() causes server crash when csv file contains too few field separators or contains empty lines - [7607](https://github.com/MonetDB/MonetDB/issues/7607): Adding a column of serial type fails with "Access denied for <user> to schema 'sys'"

1 0

Announcement: New Aug2024-SP1 Bugfix release of MonetDB suite
by Sjoerd Mullender 15 Nov '24

15 Nov '24

The MonetDB Foundation is pleased to announce the Aug2024-SP1 bugfix release of the MonetDB suite of programs. More information about MonetDB can be found on our website at <https://www.monetdb.org/>. For details on this release, please see the release notes at <https://www.monetdb.org/release-notes/aug2024/>. As usual, the download location is <https://www.monetdb.org/downloads/>. ### MonetDB5 Server - The server prints out an informative message for the case of a graceful termination. ### SQL Frontend - Improve casting to generic decimal type by choosing a better fit for precision and scale instead of defaulting to 18 and 3, respectively. - When for whatever reason the upgrade code produces an error, we now exit the server. Before the server would limp on with what is basically a broken database. - Improved the violation message of CHECK constraints when violated. It now includes the schema name of the constraint and the check clause. - Increase the buffer size used by hot snapshot from 64kiB to 1MiB, and make it configurable through setting 'hot_snapshot_buffer_size'. It must be a multiple of 512. ### Merovingian - Tweak socket parameters to simulate network activity on client connections. This prevents firewalls from killing connections that seem idle but are actually waiting for a long-running query. Can be controlled with a new 'keepalive' option to monetdbd. ### Bug Fixes - [7281](https://github.com/MonetDB/MonetDB/issues/7281): UDFs defined at compile time in a user schema should not become system functions - [7563](https://github.com/MonetDB/MonetDB/issues/7563): Unexpected result when using `IS DISTINCT FROM` in `VIEW` - [7567](https://github.com/MonetDB/MonetDB/issues/7567): creating remote table from subquery crashes the server - [7569](https://github.com/MonetDB/MonetDB/issues/7569): Column of temporary table changes when another is updated - [7570](https://github.com/MonetDB/MonetDB/issues/7570): BUG in the "str_to_timestamp" function - [7571](https://github.com/MonetDB/MonetDB/issues/7571): Crash when integer overflow in `ORDER BY` - [7572](https://github.com/MonetDB/MonetDB/issues/7572): column max length is not stored as specified and accepted at creation time - [7575](https://github.com/MonetDB/MonetDB/issues/7575): Incorrect BAT properties after mmapped BAT "leaks" to disk with restart. - [7576](https://github.com/MonetDB/MonetDB/issues/7576): unescaping UTF-16 code units goes wrong in json.text - [7577](https://github.com/MonetDB/MonetDB/issues/7577): Crash when using `CHECK` constraint - [7580](https://github.com/MonetDB/MonetDB/issues/7580): statistics optimizer handles date difference incorrectly - [7582](https://github.com/MonetDB/MonetDB/issues/7582): SIGSEGV when creating a SQL function with RETURN CASE WHEN EXISTS (..) - [7583](https://github.com/MonetDB/MonetDB/issues/7583): Query slowdown after deleting rows from large table - [7584](https://github.com/MonetDB/MonetDB/issues/7584): SO_KEEPALIVE should be configured sensibly - [7585](https://github.com/MonetDB/MonetDB/issues/7585): rel2bin_join: Assertion `sql->session->status == -10' failed. - [7587](https://github.com/MonetDB/MonetDB/issues/7587): Line/row numbers get out of sync with COPY INTO .. BEST EFFORT

2 1

help services
by agata.cooper159＠gmail.com 28 Aug '24

28 Aug '24

In today's fast-paced academic environment, students often find themselves overwhelmed with assignments and tight deadlines. When it comes to complex subjects like finance, the pressure can be even greater. This is where professional services like https://writepaper.com/finance-essay-writing-services come in handy. These services offer expert assistance in writing high-quality finance essays, helping students to manage their workload more effectively and achieve better grades.

1 0

finance essay writing services
by agata.cooper159＠gmail.com 28 Aug '24

28 Aug '24

In today's fast-paced academic environment, students often find themselves overwhelmed with assignments and tight deadlines. When it comes to complex subjects like finance, the pressure can be even greater. This is where professional services like <a href="https://writepaper.com/finance-essay-writing-services">https://writepaper.com/finance-essay-writing-services</a> come in handy. These services offer expert assistance in writing high-quality finance essays, helping students to manage their workload more effectively and achieve better grades.

1 0

Announcement: New Aug2024 Feature release of MonetDB suite
by Sjoerd Mullender 26 Aug '24

26 Aug '24

The MonetDB Foundation is pleased to announce the Aug2024 feature release of the MonetDB suite of programs. More information about MonetDB can be found on our website at <https://www.monetdb.org/>. For details on this release, please see the release notes at <https://www.monetdb.org/release-notes/aug2024/>. As usual, the download location is <https://www.monetdb.org/downloads/>. The Aug2024 documentation can be found [here](https://www.monetdb.org/documentation-Aug2024/). ## Aug2024 Feature Release (11.51.3) ### - The CMake configuration files for building extensions have now been included in the various MonetDB development RPMs and debs. - Removed upgrade code for versions before Jul2021. - The shared library (.dll aka .so files) now have the version number as part of the name. This should allow the building of compatibility versions that can be installed in parallel to the latest version. - Some of the Debian/Ubuntu packages have been renamed. The old monetdb5 names have been changed to plain monetdb, and libmonetdb5-server-* packages have been renamed monetdb-*. - The names of some of the provided RPM files have been changed. References to the old MonetDB5 name have been removed. All packages are now just MonetDB. ### Client Package - Msqldump now accepts --output and --outputdir options. When the --outputdir option is used, the dump is placed in the file dump.sql in the specified directory and all tables are dumped to separate CSV files. In this way it is feasible to edit the dump script by hand if needed, even for a large database. - The --table (-t) option of msqldump now accepts SQL-style % wildcard characters to dump all tables that match the pattern. E.g. -t %test%.%test% dumps all tables with 'test' in both the schema and table name. - Implemented interrupt handling in mclient. When using mclient interactively, an interrupt (usually control-C) stops whatever the client is doing. When editing a line, the line is discarded; when editing a second or later line of a query, the whole query is discarded; when a query is being executed, the server is asked to stop the query at its earliest convenience. Stopping a running query can only be done with an up-to-date server. All of this does not work on Windows. ### Mapi Library - Add new columns to sys.sessions. Column 'language' is usually 'sql'. Column 'peer' is the network address of the client (something like '[::1]:46558' or '<UNIX SOCKET>'). Columns 'hostname', 'application', 'client', 'clientpid' and 'remark' can be set by the client. Libmapi/mclient, pymonetdb and monetdb-java have been modified to fill in sensible default values. ### ODBC Driver - Extended ODBC Data Source Setup program on Windows with fields to specify optional Client Information such as Application Name and Client Remark. - Extended ODBC Data Source Setup program on Windows with fields to specify new TLS Connection settings. - Extended ODBC Data Source Setup program on Windows with a Test-button to quickly test connectivity to a MonetDB server. - ODBC now supports TLS. It can be configured through the following DSN- or Connection String attributes (canonical name / user friendly name): TLS / Encrypt = ON/OFF CERT / Server Certificate = PATH CERTHASH / Server Certificate Hash = sha256:HEXDIGITS CLIENTKEY / Client Key = PATH CLIENTCERT / Client Certificate = PATH - Several more connection properties have been made configurable: SCHEMA / Schema = NAME TIMEZONE / Time Zone = Minutes East Of UTC REPLYSIZE / Reply Size = NUMBER LOGFILE / Log File = PATH LOGINTIMEOUT / Login Timeout = MILLISECONDS CONNECTIONTIMEOUT / Connection Timeout = MILLISECONDS AUTOCOMMIT / Autocommit = ON/OFF SOCK / Unix Socket = PATH (unix only) - SQLBrowseConnect() adds On/Off suggestions to boolean settings and prioritizes the DATABASE attribute if it notices monetdbd requires one. Apart from that only UID/User and PWD/Password are required, all others have sensible defaults. ### Stream Library - ICONV support has been removed from the stream library. The server itself only needs UTF-8 support. The client (mclient) does have iconv support. - CURL support has been removed from the stream library. If support is needed, look at the source code in either streamcat.c or mclient.c. ### MonetDB Common - Various changes were made having to do with things like case-insensitive comparisons and converting to upper or lower case. Case insensitive comparison (i.e. the ILIKE operator) uses case folding which is similar to converting to lower case, but changes more characters, also sometimes to multiple characters (e.g. German sharp s (ß) compares equal to SS). - Made some changes to how BAT descriptors are allocated. They are now allocated in bulk, meaning fewer malloc/free calls during processing. - Removed macro BBP_cache and its associated code. Checking whether a BAT is cached (loaded in memory) can be done by checking the BBPLOADED bit in the BBP_status value. Getting a pointer to the BAT descriptor can be done by using BBP_desc. - The SQL transaction ID is no longer saved in the BBP.dir file. - Made some changes to the TIMEOUT macros. Most importantly, they now get a pointer to a QryCtx structure as argument instead of the timeout value. ### SQL Frontend - The "phash" column in the sys.storage() table now indicates whether a hash exists. If the hash is not loaded but there is a hash available on disk, the phash value is "true", but the "hashes" value is 0. - Added execution privilege on all sys.generate_series(first, limit) and sys.generate_series(first, limit, stepsize) functions to public, so all users can now call these table producing generator functions. - Extend CREATE USER MAX_MEMORY and ALTER USER MAX_MEMORY to accept strings of the form '10MiB', '10G', etc. - Extended view sys.sessions and function sys.sessions() with new columns: language, peer, hostname, application, client, clientpid and remark. - All users now have SELECT privilege on view sys.sessions, but non-admin users only see their own sessions. - Added procedure sys.setclientinfo(property string, value string) to allow the client application to set a specific client info property. - Added system table sys.clientinfo_properties that lists the supported client info properties and their associated column name in sys.sessions view. It contains property names: ClientHostname, ApplicationName, ClientLibrary, ClientPid and ClientRemark. - Extended sys.generate_series() to generate dates. Added 2 new functions: sys.generate_series(first date, "limit" date, stepsize interval month) and sys.generate_series(first date, "limit" date, stepsize interval day). - Added support for select exp, count(*) group by 1 order by 1; ie. using numeric references in group by clause. - Added support for GROUP BY ALL. This finds all expressions from the selections which aren't aggregations and groups on those. At least one aggregation must be specified. The ALL keyword can also be replaced by '*', so: GROUP BY *. - Added support for ORDER BY ALL. This orders on all columns of the selection. The ALL keyword can also be replaced by '*', so: ORDER BY *. - Removed the obsolete ANALYZE statement syntax options: SAMPLE nn and MINMAX. Both options have been ignored since release Jan2022. Now they are no longer accepted in the ANALYZE statement. - The ANALYZE statement can now be used in procedures, functions and triggers. - Make schema renaming more permissive. A schema can be renamed if it does not contain objects that are a dependency for objects outside the schema. If such dependencies exist, they are shown in the table sys.dependencies. - Add a DECIMAL AS clause to COPY INTO that configures the decimal separator and thousands separator for decimals, temporal types and floats. - Introduce IS [NOT] DISTINCT FROM syntax. The syntax allows two values to be compared. The comparison always returns boolean FALSE or TRUE never NULL. - SQL2023 feature: Introduce UNIQUE NULLS [NOT] DISTINCT syntax which allows for NULLS to be treated as unique, i.e. a column with this constraint can have one NULL value at most. - SQL2023 feature: Allow project and ORDER BY expressions on UNIQUE constrained columns when the primary key column is used in a GROUP BY expression. ### Bug Fixes - [7045](https://github.com/MonetDB/MonetDB/issues/7045): A value filtered in a subquery finds its way to a later filter in the outer query - [7097](https://github.com/MonetDB/MonetDB/issues/7097): Add an 'ANY' or 'ARBITRARY' aggregate function - [7245](https://github.com/MonetDB/MonetDB/issues/7245): monetdbe silently importing 0 rows, at random - [7265](https://github.com/MonetDB/MonetDB/issues/7265): COPY INTO Not Reading in all records of fixed width delimited file - [7272](https://github.com/MonetDB/MonetDB/issues/7272): Missed rewrite to bulk operators in simple SQL UDF - [7312](https://github.com/MonetDB/MonetDB/issues/7312): Test Button for ODBC Driver - [7332](https://github.com/MonetDB/MonetDB/issues/7332): Support IS [NOT] DISTINCT FROM predicate - [7353](https://github.com/MonetDB/MonetDB/issues/7353): INTERVAL SECOND columns become incorrect when an INTERVAL HOUR column is present - [7367](https://github.com/MonetDB/MonetDB/issues/7367): Libraries linked more than once? - [7370](https://github.com/MonetDB/MonetDB/issues/7370): timestamp_to_str function not aware of timezone - [7374](https://github.com/MonetDB/MonetDB/issues/7374): Different date and time returns - [7392](https://github.com/MonetDB/MonetDB/issues/7392): Evaluate Profile-Guided Optimization - [7424](https://github.com/MonetDB/MonetDB/issues/7424): Performance issue in select-joins - [7459](https://github.com/MonetDB/MonetDB/issues/7459): Crash when using `CONTAINS` in `ORDER BY` clause - [7460](https://github.com/MonetDB/MonetDB/issues/7460): Crash when using `CAST` and `BETWEEN AND` - [7463](https://github.com/MonetDB/MonetDB/issues/7463): Unexpected result when using `CONTAINS` and type casting - [7466](https://github.com/MonetDB/MonetDB/issues/7466): Crash when `INNER JOIN` with `CONTAINS` - [7467](https://github.com/MonetDB/MonetDB/issues/7467): Conversion TIMESTAMPTZ to TIME does not take session TZ into account - [7474](https://github.com/MonetDB/MonetDB/issues/7474): MonetDB server crashes in `VLTgenerator_table_` - [7475](https://github.com/MonetDB/MonetDB/issues/7475): MonetDB server crashes in `__nss_database_lookup` - [7476](https://github.com/MonetDB/MonetDB/issues/7476): MonetDB server crashes in `subrel_bin` - [7477](https://github.com/MonetDB/MonetDB/issues/7477): MonetDB server crashes in `atom_cmp` - [7480](https://github.com/MonetDB/MonetDB/issues/7480): MonetDB server crashes in `get_rel_count` - [7481](https://github.com/MonetDB/MonetDB/issues/7481): MonetDB server crashes in `mvc_row_result_wrap` - [7482](https://github.com/MonetDB/MonetDB/issues/7482): MonetDB server crashes in `bin_find_smallest_column` - [7483](https://github.com/MonetDB/MonetDB/issues/7483): MonetDB server crashes in `rel_get_statistics_` - [7484](https://github.com/MonetDB/MonetDB/issues/7484): MonetDB server crashes in `rel_optimize_projections_` - [7485](https://github.com/MonetDB/MonetDB/issues/7485): MonetDB server crashes in `exp_setalias` - [7486](https://github.com/MonetDB/MonetDB/issues/7486): MonetDB server crashes in `ALGgroupby` - [7488](https://github.com/MonetDB/MonetDB/issues/7488): MonetDB server crashes in `strCmp` - [7497](https://github.com/MonetDB/MonetDB/issues/7497): Multi-column IN clause with value list produces syntax error - [7500](https://github.com/MonetDB/MonetDB/issues/7500): request: ANALYZE statement should be allowed to be used in the body of a procedure or function or trigger. - [7514](https://github.com/MonetDB/MonetDB/issues/7514): Nonexistent window function raises `ParseException:SQLparser:42000!Query too complex: running out of stack space` - [7517](https://github.com/MonetDB/MonetDB/issues/7517): UNLOGGED tables don't get cleaned up properly when DROPped. - [7521](https://github.com/MonetDB/MonetDB/issues/7521): Unexpected result when using `IS DISTINCT FROM` - [7522](https://github.com/MonetDB/MonetDB/issues/7522): Crash when creating view with `HAVING` - [7523](https://github.com/MonetDB/MonetDB/issues/7523): Assertion failure when using `CONTAINS` - [7524](https://github.com/MonetDB/MonetDB/issues/7524): Unexpected error when using `NATURAL RIGHT JOIN` - [7525](https://github.com/MonetDB/MonetDB/issues/7525): Related to bug #7422 (variadic arguments in aggregate UDFs) - [7527](https://github.com/MonetDB/MonetDB/issues/7527): Unexpected result when using `IS DISTINCT FROM` with `RIGHT JOIN` - [7528](https://github.com/MonetDB/MonetDB/issues/7528): Assertion failure when using `JAROWINKLER` in `ORDER BY` clause - [7529](https://github.com/MonetDB/MonetDB/issues/7529): Finding minimum value in reverse sorted column with NULL values at the end gives wrong result. - [7530](https://github.com/MonetDB/MonetDB/issues/7530): Assertion failure when using `JAROWINKLER` with empty string - [7533](https://github.com/MonetDB/MonetDB/issues/7533): DROP of a schema with CASCADE option drops tables, but bats are not removed. - [7534](https://github.com/MonetDB/MonetDB/issues/7534): Unexpected result when using `IS DISTINCT FROM` with `AND` - [7535](https://github.com/MonetDB/MonetDB/issues/7535): Assertion failure when using `GROUP BY` when `CREATE VIEW` - [7538](https://github.com/MonetDB/MonetDB/issues/7538): BUG with decimal values - [7539](https://github.com/MonetDB/MonetDB/issues/7539): Crash when using `IS DISTINCT FROM` with `SIN` - [7540](https://github.com/MonetDB/MonetDB/issues/7540): Assertion failure when using `STARTSWITH` - [7542](https://github.com/MonetDB/MonetDB/issues/7542): Nested query triggers an assert - [7543](https://github.com/MonetDB/MonetDB/issues/7543): Unexpected result when using `IS DISTINCT FROM` with constants - [7544](https://github.com/MonetDB/MonetDB/issues/7544): Unexpected result when using `STARTSWITH` - [7545](https://github.com/MonetDB/MonetDB/issues/7545): Crash when creating view with `GROUP BY` - [7547](https://github.com/MonetDB/MonetDB/issues/7547): drop login trigger causes server crash - [7550](https://github.com/MonetDB/MonetDB/issues/7550): non-admin user can no longer query sys.statistics or information_schema.tables - [7552](https://github.com/MonetDB/MonetDB/issues/7552): Unexpected result when using `NULL` constant in comparison - [7553](https://github.com/MonetDB/MonetDB/issues/7553): Assertion failure when using `INNER JOIN` on `STARTSWITH` - [7554](https://github.com/MonetDB/MonetDB/issues/7554): Unexpected result when using range comparison with `NULL` - [7555](https://github.com/MonetDB/MonetDB/issues/7555): Unexpected result when casting integer to boolean in comparison - [7556](https://github.com/MonetDB/MonetDB/issues/7556): Assertion failure when using `STARTSWITH` with view - [7562](https://github.com/MonetDB/MonetDB/issues/7562): Assertion failure when comparing `INTERVAL` value - [7566](https://github.com/MonetDB/MonetDB/issues/7566): After a while, all new sessions (connections) may get refused

1 0

focused topics
by genaomatilda＠gmail.com 23 Aug '24

23 Aug '24

Nowadays, there are a lot of narrowly focused topics and clearly for me where were more humanitarian sciences https://studyfy.com/service/do-my-homework do my homework for me very high quality and fast, I think they would cope with your topic as well

1 0

focused topics
by genaomatilda＠gmail.com 23 Aug '24

23 Aug '24

Nowadays, there are a lot of narrowly focused topics and clearly for me where were more humanitarian sciences https://studyfy.com/service/do-my-homework do my homework for me very high quality and fast, I think they would cope with your topic as well

1 0

**Unlocking Success in Nursing Education: The Comprehensive Guide to BSN Writing Services**
by noheta1148＠morxin.com 23 Aug '24

23 Aug '24

\*\***Unlocking Success in Nursing Education: The Comprehensive Guide to BSN Writing Services\*\*** The pursuit of a Bachelor of Science in Nursing (BSN) is a commendable endeavor, marking a significant step toward a rewarding and impactful career in healthcare. However, the journey is often challenging, demanding not only practical skills and [take my online nursing class](https://bsnwritingservices.com/) clinical knowledge but also exceptional academic writing abilities. From research papers and essays to clinical case studies and capstone projects, the array of writing assignments can be overwhelming. This is where professional BSN writing services come into play, providing crucial support to nursing students striving for academic excellence. \### The Importance of Writing in BSN Programs Writing is a fundamental component of BSN programs, serving multiple purposes: 1\. \*\*Knowledge Application\*\*: Writing assignments allow students to apply theoretical knowledge to real-world scenarios, enhancing their understanding and retention of nursing concepts. 2\. \*\*Critical Thinking\*\*: Crafting well-structured papers fosters critical thinking and analytical skills, essential for clinical decision-making and problem-solving. 3\. \*\*Communication Skills\*\*: Effective writing is integral to professional communication in nursing, whether it's documenting patient care, writing reports, or engaging in interdisciplinary collaboration. 4\. \*\*Professional Development\*\*: Through reflective writing and research, students develop a deeper awareness of their practice, ethics, and professional growth. \### The Role of BSN Writing Services BSN writing services provide specialized assistance to nursing students, ensuring they can meet the rigorous academic standards of their programs while managing the demands of clinical practice and personal commitments. These services are typically offered by experienced writers with backgrounds in nursing and healthcare, who understand the intricacies of nursing education and the expectations of academic institutions. \### Benefits of BSN Writing Services 1\. \*\*Expertise and Experience\*\*: Professional writers bring a wealth of knowledge and experience in nursing, ensuring that your assignments are accurate, relevant, and insightful. They are familiar with nursing terminology, evidence-based practice, and ethical considerations. 2\. \*\*Customized Support\*\*: BSN writing services offer tailored assistance based on your specific requirements, academic level, and institutional guidelines. This personalized approach ensures that the content is aligned with your [capella 4050 assessment 1](https://bsnwritingservices.com/capella-4050-assessment-1/) writing style and academic goals. 3\. \*\*Time Management\*\*: Balancing coursework, clinical rotations, and personal life can be challenging. Writing services help alleviate this burden by delivering high-quality papers within stipulated deadlines, allowing you to focus on other critical aspects of your education. 4\. \*\*Plagiarism-Free Content\*\*: Reputable writing services guarantee original, plagiarism-free content, supported by thorough research and proper citations. This ensures your work maintains academic integrity and meets the standards of your institution. 5\. \*\*Confidentiality\*\*: Your privacy is paramount. Trustworthy BSN writing services prioritize confidentiality, protecting your personal information and academic reputation through secure communication channels and data protection measures. \### Types of BSN Writing Services BSN writing services cover a broad range of academic needs, including but not limited to: 1\. \*\*Research Papers\*\*: Comprehensive research papers involve literature reviews, data analysis, and critical evaluation of nursing studies. Professional writers can help you develop a coherent and compelling argument, supported by current evidence and scholarly sources. 2\. \*\*Case Studies\*\*: Writing detailed case studies requires an in-depth understanding of patient scenarios, diagnosis, treatment plans, and nursing interventions. Expert writers can assist in presenting these complex cases clearly and systematically. 3\. \*\*Essays and Reflection Papers\*\*: Whether it's an opinion piece, a reflective essay on clinical experiences, or a discussion on nursing ethics, writing services can help articulate your thoughts and insights effectively. 4\. \*\*Capstone Projects\*\*: Capstone projects are culminating assignments that integrate theoretical knowledge with practical application. Writing services can guide you through this comprehensive project, ensuring it showcases your mastery of nursing concepts and skills. 5\. \*\*Theses and Dissertations\*\*: For those pursuing advanced research, writing a thesis or dissertation involves extensive research, critical analysis, and original contributions to the field of nursing. Professional writers can support you at every stage, from proposal development to final editing. \### How to Choose the Right BSN Writing Service Selecting the right BSN writing service is crucial for ensuring the quality [capella 4040 assessment 2](https://bsnwritingservices.com/capella-4040-assessment-2/) and reliability of the assistance you receive. Here are key factors to consider: 1\. \*\*Reputation and Reviews\*\*: Research the service provider’s reputation by reading reviews and testimonials from past clients. Positive feedback and high ratings are indicators of reliable and high-quality service. 2\. \*\*Writer Qualifications\*\*: Ensure that the writers have relevant credentials, such as advanced degrees in nursing or related fields, and significant experience in academic writing. Qualified writers are better equipped to understand and meet the specific demands of nursing assignments. 3\. \*\*Customization and Communication\*\*: Choose a service that offers direct communication with writers and allows for customization of assignments. The ability to request revisions and provide feedback ensures that the final product meets your expectations. 4\. \*\*Pricing and Payment Options\*\*: Transparent pricing structures and secure payment methods are essential for a trustworthy service. Compare pricing among different providers to find a service that offers good value without compromising quality. 5\. \*\*Guarantees\*\*: Look for guarantees such as plagiarism-free content, confidentiality, adherence to deadlines, and satisfaction guarantees. These assurances reflect the service’s commitment to quality and customer satisfaction. \### Specialized Writing Needs in BSN Programs 1\. \*\*Evidence-Based Practice\*\*: BSN programs emphasize the importance of evidence-based practice (EBP) in nursing care. Writing services can help students analyze research studies, integrate evidence into practice, and discuss the implications for patient outcomes. This includes systematic reviews, meta-analyses, and clinical guidelines. 2\. \*\*Clinical Reflections\*\*: Reflective writing is a key component of nursing education, promoting self-awareness and professional growth. BSN writing services can assist students in articulating their clinical experiences, exploring challenges, and identifying areas for improvement. 3\. \*\*Ethical Considerations\*\*: Nursing ethics are integral to professional practice. Writing services ensure that assignments discussing ethical dilemmas, decision-making frameworks, and patient advocacy are well-researched and thoughtfully presented. \### The Process of Working with BSN Writing Services Engaging with a BSN writing service typically involves the following steps: 1\. \*\*Initial Consultation\*\*: The process begins with an initial consultation where you discuss your assignment requirements, deadlines, and any specific guidelines provided by your institution. This helps the service understand your needs and tailor [capella 4060 assessment 1](https://bsnwritingservices.com/capella-4060-assessment-1/) their assistance accordingly. 2\. \*\*Writer Assignment\*\*: Based on your requirements, the service assigns a qualified writer with relevant expertise in nursing and academic writing. You may have the option to choose your writer based on their qualifications and past work. 3\. \*\*Research and Writing\*\*: The writer conducts thorough research, utilizing credible sources to develop a well-structured and coherent paper. They ensure that the content is original, accurate, and aligns with your academic goals. 4\. \*\*Draft Review and Feedback\*\*: You receive a draft of the assignment for review. This allows you to provide feedback, request revisions, and ensure that the paper meets your expectations. Direct communication with the writer facilitates this collaborative process. 5\. \*\*Final Delivery\*\*: Once all revisions are made and you are satisfied with the final product, the completed assignment is delivered to you. The service ensures that the paper is properly formatted, cited, and ready for submission. \### Conclusion: Investing in Your Academic and Professional Future In conclusion, professional BSN writing services offer invaluable support to nursing students navigating the complexities of their academic journey. By leveraging the expertise of qualified writers, you can enhance your understanding of nursing concepts, improve your writing skills, and achieve academic success. Whether you're facing challenges with complex assignments, struggling with time management, or seeking to elevate your academic performance, BSN writing services provide practical solutions tailored to your educational needs. Choosing a reputable BSN writing service ensures that you receive high-quality, customized assistance that aligns with your academic goals and career aspirations. Embrace the opportunity to collaborate with experts who share your passion for nursing and are committed to your success in the field. By investing in BSN writing services, you’re not just completing assignments; you’re investing in your future as a competent and compassionate nursing professional. Seize the opportunity to excel in your BSN program, contribute to evidence-based practice, and make a positive impact in healthcare. With the right support and guidance, you can achieve your educational goals and embark on a rewarding career dedicated to improving patient outcomes and advancing the field of nursing. \--- This extended content provides a comprehensive overview of BSN writing services, emphasizing their benefits, specialized services, and considerations for choosing the right provider, while also detailing the process of working with such services to support nursing students' academic and professional development.

2 1

*Unlocking Success in Nursing Education: The Comprehensive Guide to BSN Writing Services**
by noheta1148＠morxin.com 11 Aug '24

11 Aug '24

**Unlocking Success in Nursing Education: The Comprehensive Guide to BSN Writing Services** The pursuit of a Bachelor of Science in Nursing (BSN) is a commendable endeavor, marking a significant step toward a rewarding and impactful career in healthcare. However, the journey is often challenging, demanding not only practical skills and <a href="https://bsnwritingservices.com/">take my online nursing class</a> clinical knowledge but also exceptional academic writing abilities. From research papers and essays to clinical case studies and capstone projects, the array of writing assignments can be overwhelming. This is where professional BSN writing services come into play, providing crucial support to nursing students striving for academic excellence. ### The Importance of Writing in BSN Programs Writing is a fundamental component of BSN programs, serving multiple purposes: 1. **Knowledge Application**: Writing assignments allow students to apply theoretical knowledge to real-world scenarios, enhancing their understanding and retention of nursing concepts. 2. **Critical Thinking**: Crafting well-structured papers fosters critical thinking and analytical skills, essential for clinical decision-making and problem-solving. 3. **Communication Skills**: Effective writing is integral to professional communication in nursing, whether it's documenting patient care, writing reports, or engaging in interdisciplinary collaboration. 4. **Professional Development**: Through reflective writing and research, students develop a deeper awareness of their practice, ethics, and professional growth. ### The Role of BSN Writing Services BSN writing services provide specialized assistance to nursing students, ensuring they can meet the rigorous academic standards of their programs while managing the demands of clinical practice and personal commitments. These services are typically offered by experienced writers with backgrounds in nursing and healthcare, who understand the intricacies of nursing education and the expectations of academic institutions. ### Benefits of BSN Writing Services 1. **Expertise and Experience**: Professional writers bring a wealth of knowledge and experience in nursing, ensuring that your assignments are accurate, relevant, and insightful. They are familiar with nursing terminology, evidence-based practice, and ethical considerations. 2. **Customized Support**: BSN writing services offer tailored assistance based on your specific requirements, academic level, and institutional guidelines. This personalized approach ensures that the content is aligned with your <a href="https://bsnwritingservices.com/capella-4050-assessment-1/">capella 4050 assessment 1</a> writing style and academic goals. 3. **Time Management**: Balancing coursework, clinical rotations, and personal life can be challenging. Writing services help alleviate this burden by delivering high-quality papers within stipulated deadlines, allowing you to focus on other critical aspects of your education. 4. **Plagiarism-Free Content**: Reputable writing services guarantee original, plagiarism-free content, supported by thorough research and proper citations. This ensures your work maintains academic integrity and meets the standards of your institution. 5. **Confidentiality**: Your privacy is paramount. Trustworthy BSN writing services prioritize confidentiality, protecting your personal information and academic reputation through secure communication channels and data protection measures. ### Types of BSN Writing Services BSN writing services cover a broad range of academic needs, including but not limited to: 1. **Research Papers**: Comprehensive research papers involve literature reviews, data analysis, and critical evaluation of nursing studies. Professional writers can help you develop a coherent and compelling argument, supported by current evidence and scholarly sources. 2. **Case Studies**: Writing detailed case studies requires an in-depth understanding of patient scenarios, diagnosis, treatment plans, and nursing interventions. Expert writers can assist in presenting these complex cases clearly and systematically. 3. **Essays and Reflection Papers**: Whether it's an opinion piece, a reflective essay on clinical experiences, or a discussion on nursing ethics, writing services can help articulate your thoughts and insights effectively. 4. **Capstone Projects**: Capstone projects are culminating assignments that integrate theoretical knowledge with practical application. Writing services can guide you through this comprehensive project, ensuring it showcases your mastery of nursing concepts and skills. 5. **Theses and Dissertations**: For those pursuing advanced research, writing a thesis or dissertation involves extensive research, critical analysis, and original contributions to the field of nursing. Professional writers can support you at every stage, from proposal development to final editing. ### How to Choose the Right BSN Writing Service Selecting the right BSN writing service is crucial for ensuring the quality <a href="https://bsnwritingservices.com/capella-4040-assessment-2/">capella 4040 assessment 2</a> and reliability of the assistance you receive. Here are key factors to consider: 1. **Reputation and Reviews**: Research the service provider’s reputation by reading reviews and testimonials from past clients. Positive feedback and high ratings are indicators of reliable and high-quality service. 2. **Writer Qualifications**: Ensure that the writers have relevant credentials, such as advanced degrees in nursing or related fields, and significant experience in academic writing. Qualified writers are better equipped to understand and meet the specific demands of nursing assignments. 3. **Customization and Communication**: Choose a service that offers direct communication with writers and allows for customization of assignments. The ability to request revisions and provide feedback ensures that the final product meets your expectations. 4. **Pricing and Payment Options**: Transparent pricing structures and secure payment methods are essential for a trustworthy service. Compare pricing among different providers to find a service that offers good value without compromising quality. 5. **Guarantees**: Look for guarantees such as plagiarism-free content, confidentiality, adherence to deadlines, and satisfaction guarantees. These assurances reflect the service’s commitment to quality and customer satisfaction. ### Specialized Writing Needs in BSN Programs 1. **Evidence-Based Practice**: BSN programs emphasize the importance of evidence-based practice (EBP) in nursing care. Writing services can help students analyze research studies, integrate evidence into practice, and discuss the implications for patient outcomes. This includes systematic reviews, meta-analyses, and clinical guidelines. 2. **Clinical Reflections**: Reflective writing is a key component of nursing education, promoting self-awareness and professional growth. BSN writing services can assist students in articulating their clinical experiences, exploring challenges, and identifying areas for improvement. 3. **Ethical Considerations**: Nursing ethics are integral to professional practice. Writing services ensure that assignments discussing ethical dilemmas, decision-making frameworks, and patient advocacy are well-researched and thoughtfully presented. ### The Process of Working with BSN Writing Services Engaging with a BSN writing service typically involves the following steps: 1. **Initial Consultation**: The process begins with an initial consultation where you discuss your assignment requirements, deadlines, and any specific guidelines provided by your institution. This helps the service understand your needs and tailor <a href="https://bsnwritingservices.com/capella-4060-assessment-1/">capella 4060 assessment 1</a> their assistance accordingly. 2. **Writer Assignment**: Based on your requirements, the service assigns a qualified writer with relevant expertise in nursing and academic writing. You may have the option to choose your writer based on their qualifications and past work. 3. **Research and Writing**: The writer conducts thorough research, utilizing credible sources to develop a well-structured and coherent paper. They ensure that the content is original, accurate, and aligns with your academic goals. 4. **Draft Review and Feedback**: You receive a draft of the assignment for review. This allows you to provide feedback, request revisions, and ensure that the paper meets your expectations. Direct communication with the writer facilitates this collaborative process. 5. **Final Delivery**: Once all revisions are made and you are satisfied with the final product, the completed assignment is delivered to you. The service ensures that the paper is properly formatted, cited, and ready for submission. ### Conclusion: Investing in Your Academic and Professional Future In conclusion, professional BSN writing services offer invaluable support to nursing students navigating the complexities of their academic journey. By leveraging the expertise of qualified writers, you can enhance your understanding of nursing concepts, improve your writing skills, and achieve academic success. Whether you're facing challenges with complex assignments, struggling with time management, or seeking to elevate your academic performance, BSN writing services provide practical solutions tailored to your educational needs. Choosing a reputable BSN writing service ensures that you receive high-quality, customized assistance that aligns with your academic goals and career aspirations. Embrace the opportunity to collaborate with experts who share your passion for nursing and are committed to your success in the field. By investing in BSN writing services, you’re not just completing assignments; you’re investing in your future as a competent and compassionate nursing professional. Seize the opportunity to excel in your BSN program, contribute to evidence-based practice, and make a positive impact in healthcare. With the right support and guidance, you can achieve your educational goals and embark on a rewarding career dedicated to improving patient outcomes and advancing the field of nursing. --- This extended content provides a comprehensive overview of BSN writing services, emphasizing their benefits, specialized services, and considerations for choosing the right provider, while also detailing the process of working with such services to support nursing students' academic and professional development.

2 1

Announcement: New Dec2023-SP4 Bugfix release of MonetDB suite
by Sjoerd Mullender 09 Jul '24

09 Jul '24

The MonetDB Foundation is pleased to announce the Dec2023-SP4 bugfix release of the MonetDB suite of programs. More information about MonetDB can be found on our website at <https://www.monetdb.org/>. For details on this release, please see the release notes at <https://www.monetdb.org/release-notes/dec2023/>. As usual, the download location is <https://www.monetdb.org/downloads/>. ### MonetDB5 Server - The mserver5 program has a new option: --without-geom. If the server was compiled with geom support (or the geom module was installed in a binary distribution), this option allows the server to start without the geom module, so that it can proceed with a database that was created without geom. ### SQL Frontend - When sys.persist_unlogged is called for a table, it may return that zero rows were persisted. If this is because the call was done too early, i.e. the table was recently created and the write-ahead log where this was logged has not been processed yet, the call will request an immediate write-ahead log rotation. This means that the WAL will be processed as soon as possible and a new call to sys.persist_unlogged soon after will likely return a positive result. ### Bug Fixes - [7509](https://github.com/MonetDB/MonetDB/issues/7509): MonetDB Dec2023-SP2 crashes at `sql_init_subtype` - [7511](https://github.com/MonetDB/MonetDB/issues/7511): MonetDB Dec2023-SP2 crashes with the `PASSWORD_HASH` function - [7512](https://github.com/MonetDB/MonetDB/issues/7512): MonetDB Dec2023-SP2 crashes with the `GET_MERGE_TABLE_PARTITION_EXPRESSIONS` function - [7513](https://github.com/MonetDB/MonetDB/issues/7513): MonetDB Dec2023-SP2 crashes with the `GETHOST` function - [7518](https://github.com/MonetDB/MonetDB/issues/7518): mserver reports errors when starting if geom module is enabled but database was created without - [7526](https://github.com/MonetDB/MonetDB/issues/7526): deadlock, causing new connections to hang indefinitely - [7531](https://github.com/MonetDB/MonetDB/issues/7531): loading more than 2147483647 rows gives issue. - [7536](https://github.com/MonetDB/MonetDB/issues/7536): Truncated file when dumping a table from mclient into a gzipped file - [7537](https://github.com/MonetDB/MonetDB/issues/7537): MonetDB crashes with a SIGSEGV due to a null pointer dereference when using prepared statements - [7541](https://github.com/MonetDB/MonetDB/issues/7541): Unexpected result when using `LEVENSHTEIN` - [7546](https://github.com/MonetDB/MonetDB/issues/7546): monetdbd leaks file descriptors when starting mserver5.

1 0

**Unlocking Success in Nursing Education: The Comprehensive Guide to BSN Writing Services**
by noheta1148＠morxin.com 15 Jun '24

15 Jun '24

**Unlocking Success in Nursing Education: The Comprehensive Guide to BSN Writing Services** The pursuit of a Bachelor of Science in Nursing (BSN) is a commendable endeavor, marking a significant step toward a rewarding and impactful career in healthcare. However, the journey is often challenging, demanding not only practical skills and <a href="https://bsnwritingservices.com/">take my online nursing class</a> clinical knowledge but also exceptional academic writing abilities. From research papers and essays to clinical case studies and capstone projects, the array of writing assignments can be overwhelming. This is where professional BSN writing services come into play, providing crucial support to nursing students striving for academic excellence. ### The Importance of Writing in BSN Programs Writing is a fundamental component of BSN programs, serving multiple purposes: 1. **Knowledge Application**: Writing assignments allow students to apply theoretical knowledge to real-world scenarios, enhancing their understanding and retention of nursing concepts. 2. **Critical Thinking**: Crafting well-structured papers fosters critical thinking and analytical skills, essential for clinical decision-making and problem-solving. 3. **Communication Skills**: Effective writing is integral to professional communication in nursing, whether it's documenting patient care, writing reports, or engaging in interdisciplinary collaboration. 4. **Professional Development**: Through reflective writing and research, students develop a deeper awareness of their practice, ethics, and professional growth. ### The Role of BSN Writing Services BSN writing services provide specialized assistance to nursing students, ensuring they can meet the rigorous academic standards of their programs while managing the demands of clinical practice and personal commitments. These services are typically offered by experienced writers with backgrounds in nursing and healthcare, who understand the intricacies of nursing education and the expectations of academic institutions. ### Benefits of BSN Writing Services 1. **Expertise and Experience**: Professional writers bring a wealth of knowledge and experience in nursing, ensuring that your assignments are accurate, relevant, and insightful. They are familiar with nursing terminology, evidence-based practice, and ethical considerations. 2. **Customized Support**: BSN writing services offer tailored assistance based on your specific requirements, academic level, and institutional guidelines. This personalized approach ensures that the content is aligned with your <a href="https://bsnwritingservices.com/capella-4050-assessment-1/">capella 4050 assessment 1</a> writing style and academic goals. 3. **Time Management**: Balancing coursework, clinical rotations, and personal life can be challenging. Writing services help alleviate this burden by delivering high-quality papers within stipulated deadlines, allowing you to focus on other critical aspects of your education. 4. **Plagiarism-Free Content**: Reputable writing services guarantee original, plagiarism-free content, supported by thorough research and proper citations. This ensures your work maintains academic integrity and meets the standards of your institution. 5. **Confidentiality**: Your privacy is paramount. Trustworthy BSN writing services prioritize confidentiality, protecting your personal information and academic reputation through secure communication channels and data protection measures. ### Types of BSN Writing Services BSN writing services cover a broad range of academic needs, including but not limited to: 1. **Research Papers**: Comprehensive research papers involve literature reviews, data analysis, and critical evaluation of nursing studies. Professional writers can help you develop a coherent and compelling argument, supported by current evidence and scholarly sources. 2. **Case Studies**: Writing detailed case studies requires an in-depth understanding of patient scenarios, diagnosis, treatment plans, and nursing interventions. Expert writers can assist in presenting these complex cases clearly and systematically. 3. **Essays and Reflection Papers**: Whether it's an opinion piece, a reflective essay on clinical experiences, or a discussion on nursing ethics, writing services can help articulate your thoughts and insights effectively. 4. **Capstone Projects**: Capstone projects are culminating assignments that integrate theoretical knowledge with practical application. Writing services can guide you through this comprehensive project, ensuring it showcases your mastery of nursing concepts and skills. 5. **Theses and Dissertations**: For those pursuing advanced research, writing a thesis or dissertation involves extensive research, critical analysis, and original contributions to the field of nursing. Professional writers can support you at every stage, from proposal development to final editing. ### How to Choose the Right BSN Writing Service Selecting the right BSN writing service is crucial for ensuring the quality <a href="https://bsnwritingservices.com/capella-4040-assessment-2/">capella 4040 assessment 2</a> and reliability of the assistance you receive. Here are key factors to consider: 1. **Reputation and Reviews**: Research the service provider’s reputation by reading reviews and testimonials from past clients. Positive feedback and high ratings are indicators of reliable and high-quality service. 2. **Writer Qualifications**: Ensure that the writers have relevant credentials, such as advanced degrees in nursing or related fields, and significant experience in academic writing. Qualified writers are better equipped to understand and meet the specific demands of nursing assignments. 3. **Customization and Communication**: Choose a service that offers direct communication with writers and allows for customization of assignments. The ability to request revisions and provide feedback ensures that the final product meets your expectations. 4. **Pricing and Payment Options**: Transparent pricing structures and secure payment methods are essential for a trustworthy service. Compare pricing among different providers to find a service that offers good value without compromising quality. 5. **Guarantees**: Look for guarantees such as plagiarism-free content, confidentiality, adherence to deadlines, and satisfaction guarantees. These assurances reflect the service’s commitment to quality and customer satisfaction. ### Specialized Writing Needs in BSN Programs 1. **Evidence-Based Practice**: BSN programs emphasize the importance of evidence-based practice (EBP) in nursing care. Writing services can help students analyze research studies, integrate evidence into practice, and discuss the implications for patient outcomes. This includes systematic reviews, meta-analyses, and clinical guidelines. 2. **Clinical Reflections**: Reflective writing is a key component of nursing education, promoting self-awareness and professional growth. BSN writing services can assist students in articulating their clinical experiences, exploring challenges, and identifying areas for improvement. 3. **Ethical Considerations**: Nursing ethics are integral to professional practice. Writing services ensure that assignments discussing ethical dilemmas, decision-making frameworks, and patient advocacy are well-researched and thoughtfully presented. ### The Process of Working with BSN Writing Services Engaging with a BSN writing service typically involves the following steps: 1. **Initial Consultation**: The process begins with an initial consultation where you discuss your assignment requirements, deadlines, and any specific guidelines provided by your institution. This helps the service understand your needs and tailor <a href="https://bsnwritingservices.com/capella-4060-assessment-1/">capella 4060 assessment 1</a> their assistance accordingly. 2. **Writer Assignment**: Based on your requirements, the service assigns a qualified writer with relevant expertise in nursing and academic writing. You may have the option to choose your writer based on their qualifications and past work. 3. **Research and Writing**: The writer conducts thorough research, utilizing credible sources to develop a well-structured and coherent paper. They ensure that the content is original, accurate, and aligns with your academic goals. 4. **Draft Review and Feedback**: You receive a draft of the assignment for review. This allows you to provide feedback, request revisions, and ensure that the paper meets your expectations. Direct communication with the writer facilitates this collaborative process. 5. **Final Delivery**: Once all revisions are made and you are satisfied with the final product, the completed assignment is delivered to you. The service ensures that the paper is properly formatted, cited, and ready for submission. ### Conclusion: Investing in Your Academic and Professional Future In conclusion, professional BSN writing services offer invaluable support to nursing students navigating the complexities of their academic journey. By leveraging the expertise of qualified writers, you can enhance your understanding of nursing concepts, improve your writing skills, and achieve academic success. Whether you're facing challenges with complex assignments, struggling with time management, or seeking to elevate your academic performance, BSN writing services provide practical solutions tailored to your educational needs. Choosing a reputable BSN writing service ensures that you receive high-quality, customized assistance that aligns with your academic goals and career aspirations. Embrace the opportunity to collaborate with experts who share your passion for nursing and are committed to your success in the field. By investing in BSN writing services, you’re not just completing assignments; you’re investing in your future as a competent and compassionate nursing professional. Seize the opportunity to excel in your BSN program, contribute to evidence-based practice, and make a positive impact in healthcare. With the right support and guidance, you can achieve your educational goals and embark on a rewarding career dedicated to improving patient outcomes and advancing the field of nursing. --- This extended content provides a comprehensive overview of BSN writing services, emphasizing their benefits, specialized services, and considerations for choosing the right provider, while also detailing the process of working with such services to support nursing students' academic and professional development.

1 0

Announcement: New Dec2023-SP3 Bugfix release of MonetDB suite
by Sjoerd Mullender 10 May '24

10 May '24

The MonetDB Foundation is pleased to announce the Dec2023-SP3 bugfix release of the MonetDB suite of programs. More information about MonetDB can be found on our website at <https://www.monetdb.org/>. For details on this release, please see the release notes at <https://www.monetdb.org/release-notes/dec2023/>. As usual, the download location is <https://www.monetdb.org/downloads/>. ### Bug Fixes - [7422](https://github.com/MonetDB/MonetDB/issues/7422): Aggregate functions with variadic arguments - [7472](https://github.com/MonetDB/MonetDB/issues/7472): MonetDB server crashes in `tail_type` - [7473](https://github.com/MonetDB/MonetDB/issues/7473): MonetDB server crashes in `SQLunionfunc` - [7478](https://github.com/MonetDB/MonetDB/issues/7478): MonetDB server crashes in `exp_equal` - [7496](https://github.com/MonetDB/MonetDB/issues/7496): Query on view fails to produce a resultset. Assertion triggered in rel2bin_select. - [7499](https://github.com/MonetDB/MonetDB/issues/7499): create schema + set schema inside a transaction that is rolled back causes the connection to be aborted - [7501](https://github.com/MonetDB/MonetDB/issues/7501): files remain in backup causing problems at restart - [7503](https://github.com/MonetDB/MonetDB/issues/7503): MonetDB server crashes using `WHEN MATCHED THEN UPDATE` - [7504](https://github.com/MonetDB/MonetDB/issues/7504): possible deadlock when a bat is made persistent when it is also getting unloaded - [7506](https://github.com/MonetDB/MonetDB/issues/7506): MonetDB Dec2023-SP2 crashes at `rel_value_exp2` - [7507](https://github.com/MonetDB/MonetDB/issues/7507): BBPextend: ERROR: trying to extend BAT pool beyond the limit (163840000) - [7508](https://github.com/MonetDB/MonetDB/issues/7508): MonetDB Dec2023-SP2 crashes at `exp_ref`

1 0

Announcement: New Dec2023-SP2 Bugfix release of MonetDB suite
by Sjoerd Mullender 15 Apr '24

15 Apr '24

The MonetDB Foundation is pleased to announce the Dec2023-SP2 bugfix release of the MonetDB suite of programs. More information about MonetDB can be found on our website at <https://www.monetdb.org/>. For details on this release, please see the release notes at <https://www.monetdb.org/release-notes/dec2023/>. As usual, the download location is <https://www.monetdb.org/downloads/>. ### MonetDB Common - Threads have their own list of free bats. The list was not returned to the system when a thread exited, meaning that the free bats that were in the list would not be reused by any thread. This has been fixed. - Fixed a couple of deadlock situations, one actually observed, one never observed. ### MonetDB5 Server - Fixed interaction between mserver5 and remote mserver5 when only one of the two has 128 bit integer support. ### SQL Frontend - Fixed issue where equal column aliases were created. When those aliases were parsed on the remote side it could give crashes. ### Bug Fixes - [7469](https://github.com/MonetDB/MonetDB/issues/7469): Crash when using `CONTAINS` - [7479](https://github.com/MonetDB/MonetDB/issues/7479): MonetDB server crashes in `exp_ref` - [7490](https://github.com/MonetDB/MonetDB/issues/7490): commonTerms optimizer no longer works - [7495](https://github.com/MonetDB/MonetDB/issues/7495): Crash when simultaneously querying and updating a string column.

1 0

Announcement: New Dec2023-SP1 Bugfix release of MonetDB suite
by Sjoerd Mullender 18 Mar '24

18 Mar '24

The MonetDB Foundation is pleased to announce the Dec2023-SP1 bugfix release of the MonetDB suite of programs. More information about MonetDB can be found on our website at <https://www.monetdb.org/>. For details on this release, please see the release notes at <https://www.monetdb.org/release-notes/dec2023/>. As usual, the download location is <https://www.monetdb.org/downloads/>. ### - The copyright for the MonetDB software has been transferred to the newly established MonetDB Foundation, a not-for-profit foundation with the express goal of furthering the MonetDB database system. The license for the software does not change: MonetDB remains fully open source. ### Client Package - Fixed an issue where mclient wouldn't exit if the server it had connected to exited for whatever reason while the client was waiting for a query result. ### MonetDB Common - The internal hash function for floating point types has been changed. It is now no longer based on the bit representation, but on the value, meaning that +0 and -0 (yes, they both exist in floating point) now hash to the same value. - Fixed a regression where bats weren't always cleaned up when they weren't needed anymore. In particular, after a DELETE FROM table query without a WHERE clause (which deletes all rows from the table), the bats for the table get replaced by new ones, and the old, now unused, bats weren't removed from the database. ### Geom Module - We switched over to using the reentrant interface of the geos library. This fixed a number of bugs that would occur sporadically. ### SQL Frontend - performance improvement of 'startswith' and 'endswith' filter functions for join operators - The function json.isvalid(json) incorrectly returned true if the argument was null. It should return null. ### Bug Fixes - [6800](https://github.com/MonetDB/MonetDB/issues/6800): Please add information_schema (ANSI SQL norm) - [7152](https://github.com/MonetDB/MonetDB/issues/7152): Occasional dbfarm corruption upon database restart - [7390](https://github.com/MonetDB/MonetDB/issues/7390): Some MonetDB Server crashes found - [7412](https://github.com/MonetDB/MonetDB/issues/7412): MonetDB server crashes in `vscanf` - [7415](https://github.com/MonetDB/MonetDB/issues/7415): MonetDB server crashes in `HEAP_malloc` - [7416](https://github.com/MonetDB/MonetDB/issues/7416): MonetDB server crashes in `atom_get_int` - [7417](https://github.com/MonetDB/MonetDB/issues/7417): MonetDB server crashes in `trimchars`. - [7418](https://github.com/MonetDB/MonetDB/issues/7418): MonetDB server crashes in `bind_col_exp` - [7420](https://github.com/MonetDB/MonetDB/issues/7420): Performance issue with lower(string) - [7425](https://github.com/MonetDB/MonetDB/issues/7425): The last statement, execution error, is a false positive? - [7426](https://github.com/MonetDB/MonetDB/issues/7426): Unexpected result for INNER JOIN with IS NOT NULL - [7428](https://github.com/MonetDB/MonetDB/issues/7428): Unexpected result when using BETWEEN operator - [7429](https://github.com/MonetDB/MonetDB/issues/7429): Unexpected result when using `CASE WHEN` - [7430](https://github.com/MonetDB/MonetDB/issues/7430): Unexpected result when using `AND` and `IS NOT NULL` - [7431](https://github.com/MonetDB/MonetDB/issues/7431): [bug] Error code found, please confirm - [7432](https://github.com/MonetDB/MonetDB/issues/7432): MonetDB server crashes in `dameraulevenshtein` - [7433](https://github.com/MonetDB/MonetDB/issues/7433): MonetDB server crashes in `exp_atom` - [7434](https://github.com/MonetDB/MonetDB/issues/7434): MonetDB server crashes in `exp_bin` - [7435](https://github.com/MonetDB/MonetDB/issues/7435): MonetDB server crashes in `exp_copy` - [7436](https://github.com/MonetDB/MonetDB/issues/7436): MonetDB server crashes in `exp_ref` - [7437](https://github.com/MonetDB/MonetDB/issues/7437): MonetDB server crashes in `exp_values_set_supertype` - [7438](https://github.com/MonetDB/MonetDB/issues/7438): MonetDB server crashes in `exps_bind_column` - [7439](https://github.com/MonetDB/MonetDB/issues/7439): MonetDB server crashes in `exps_card` - [7440](https://github.com/MonetDB/MonetDB/issues/7440): MonetDB server crashes in `gc_col` - [7441](https://github.com/MonetDB/MonetDB/issues/7441): MonetDB server crashes in `is_column_unique` - [7442](https://github.com/MonetDB/MonetDB/issues/7442): MonetDB server crashes in `mat_join2` - [7443](https://github.com/MonetDB/MonetDB/issues/7443): MonetDB server crashes in `merge_table_prune_and_unionize` - [7444](https://github.com/MonetDB/MonetDB/issues/7444): [bug] the table cannot be created because the reserved word is incorrectly set - [7447](https://github.com/MonetDB/MonetDB/issues/7447): Unexpected result when using `BETWEEN` in `INNER JOIN` - [7448](https://github.com/MonetDB/MonetDB/issues/7448): Unexpected result when using `AND`/`OR` chain - [7450](https://github.com/MonetDB/MonetDB/issues/7450): Unexpected result when `CREATE VIEW` with `WHERE NULL` - [7451](https://github.com/MonetDB/MonetDB/issues/7451): Unexpected result when using `BETWEEN` and `CAST` - [7453](https://github.com/MonetDB/MonetDB/issues/7453): Cannot recover an msqldump - [7455](https://github.com/MonetDB/MonetDB/issues/7455): Unexpected result when using `BETWEEN` with `BOOLEAN` values - [7456](https://github.com/MonetDB/MonetDB/issues/7456): Crash when `INNER JOIN` with `VIEW` - [7457](https://github.com/MonetDB/MonetDB/issues/7457): Unexpected result when using `AND` with `INTEGER` - [7458](https://github.com/MonetDB/MonetDB/issues/7458): Unexpected result when using `SIGN` - [7461](https://github.com/MonetDB/MonetDB/issues/7461): Crash by potentially use of bad escape characters - [7462](https://github.com/MonetDB/MonetDB/issues/7462): Crash when using `BETWEEN AND` - [7465](https://github.com/MonetDB/MonetDB/issues/7465): Unexpected result when using `NULL` in `BETWEEN`

1 0

Announcement: New Dec2023 Feature release of MonetDB suite
by Sjoerd Mullender 28 Dec '23

28 Dec '23

The MonetDB team at MonetDB Solutions BV is pleased to announce the Dec2023 feature release of the MonetDB suite of programs. More information about MonetDB can be found on our website at <https://www.monetdb.org/>. For details on this release, please see the release notes at <https://www.monetdb.org/release-notes/dec2023/>. As usual, the download location is <https://www.monetdb.org/downloads/>. The Dec2023 documentation can be found [here](https://www.monetdb.org/documentation-Dec2023/). ## Dec2023 Feature Release (11.49.1) ### - All binary packages are now signed with a new key with key fingerprint DBCE 5625 94D7 1959 7B54 CE85 3F1A D47F 5521 A603. - The ranges of merge partitions are now pushed down into the low level GDK operations, giving them a handle to sometimes execute more efficiently. - Removed the PYTHON MAP external language option, as after a fork the synchronization primitives could be in any state, leading to deadlocks. During the upgrade function definitions will fallback to the normal PYTHON language option. - Implemented direct masking for strimp construction. The strimps datastructure now keeps an array of 65K 64-bit integers that is zero everywhere except at the indexes that correspond to header pairs. The entry for the nth pair in order has the nth bit of the bitstring on. These can be used to quickly construct bitstrings. ### ODBC Driver - Corrected the output value of column CHAR_OCTET_LENGTH of ODBC functions SQLColumns() and SQLProcedureColumns(). ### MonetDB Common - Fixed a (rare) race condition between copying a bat (COLcopy) and updates happening in parallel to that same bat. This may only be an actual problem with string bats, and then only in very particular circumstances. - Introduced options wal_max_dropped, wal_max_file_age and wal_max_file_size that control the write-ahead log file rotation. - Removed function BATroles to set column names on BATs. - Removed the compiled-in limit on the number of threads that can be used. The number of threads are still limited, but the limit is dictated solely by the operating system and the availability of enough memory. - We now prevent accidental upgrades from a database without 128 bit integers to one with 128 bit integers (also known as HUGEINT) from happening. Upgrades will only be done if the server is started with the option --set allow_hge_upgrade=yes. ### Geom Module - Because recent changes to the geom module require the use of geos 3.10, the geom module is no longer available in older versions of Debian and Ubuntu. Specifically, Debian 10 and 11 (buster and bullseye) and Ubuntu 20.04 (Focal Fossa) are affected. There is no automatic upgrade available for databases that were geom enabled to databases that are not, so dump + restore is the only option (if no geom types are actually used). - Implements Rtree index in GDK layer based on librtree. The index is used in the implementation of the filter functions ST_Intersects and ST_Dwithin for geometric points. - Improves shapefile support by replacing functions SHPattach, SHPpartialimport, ahd SHPimport with SHPload. - Introduces functions ST_DistanceGeographic, ST_DwithinGeographic, ST_IntersectsGeographic, ST_CoversGeographic, ST_Collects with geodesic semantics. ST_Transform can be used to convert geodetic into geographic data using libPROJ. ### MonetDB5 Server - Removed MAL functions bat.reuse and bat.reuseMap. - The MAL functions io.import and io.export have been removed. - Removed MAL functions bat.getRole and bat.setColumn since the underlying function BATroles was removed. - Change how json is stored in the database: We now normalize json strings after parsing, removing whitespace and eliminating duplicate keys in objects. - The function json.filter now properly returns json scalars instead of wrapping them in an array. - Removed the MAL tokenizer module. It was never usable from SQL and in this form never would be. ### SQL Frontend - Introduction of table returning function `persist_unlogged(schema string, table string)` that attempts to persist data in disk if "schema"."table" is unlogged table in insert only mode. If persist attempt is successful, the count of the persisted rows is returned, otherwise the count is 0. - Added ISO/IEC 9075-11 SQL/Schemata (SQL:2011) with SQL system views: information_schema.schemata information_schema.tables information_schema.views information_schema.columns information_schema.character_sets information_schema.check_constraints information_schema.table_constraints information_schema.referential_constraints information_schema.routines information_schema.parameters information_schema.sequences For details see https://www.monetdb.org/documentation/user-guide/sql-catalog/information_sc… Most views have been extended (after the standard columns) with MonetDB specific information columns such as schema_id, table_id, column_id, etc. This simplifies filtering and joins with system tables/views in sys schema when needed. Note: MonetDB does NOT support catalog qualifiers in object names, so all the CATALOG columns in these information_schema views will always return NULL. - Added support for generated column syntax: GENERATED BY DEFAULT AS IDENTITY ... This allows the user to override the default generated sequence value during inserts. - Added SQL support for: <result offset clause> and <fetch first clause> in <query expression> ::= [ <with clause> ] <query expression body> [ <order by clause> ] [ <result offset clause> ] [ <fetch first clause> ] [ <sample clause> ] <result offset clause> ::= OFFSET <offset row count> [ {ROW|ROWS} ] <fetch first clause> ::= FETCH {FIRST|NEXT} <fetch first row count> {ROW|ROWS} ONLY ### Bug Fixes - [6933](https://github.com/MonetDB/MonetDB/issues/6933): Add support for scalar function IFNULL(expr1, expr2) - [7044](https://github.com/MonetDB/MonetDB/issues/7044): Improve error message regarding 3-level SQL names - [7261](https://github.com/MonetDB/MonetDB/issues/7261): Misleading error message - [7274](https://github.com/MonetDB/MonetDB/issues/7274): Aggregate function ST_Collect crashes mserver5 - [7376](https://github.com/MonetDB/MonetDB/issues/7376): Concurrency Issue: Second Python UDF Awaits Completion of First UDF - [7391](https://github.com/MonetDB/MonetDB/issues/7391): SQL 2023 : greatest/least functions with unlimited arguments (not only 2) - [7403](https://github.com/MonetDB/MonetDB/issues/7403): Join not recognized between two row_number() columns - [7413](https://github.com/MonetDB/MonetDB/issues/7413): MonetDB server crashes in `BATcalcbetween_intern`

1 0

Announcement: New Jun2023-SP3 Bugfix release of MonetDB suite
by Sjoerd Mullender 20 Nov '23

20 Nov '23

The MonetDB team at MonetDB Solutions BV is pleased to announce the Jun2023-SP3 bugfix release of the MonetDB suite of programs. More information about MonetDB can be found on our website at <https://www.monetdb.org/>. For details on this release, please see the release notes at <https://www.monetdb.org/release-notes/jun2023/>. As usual, the download location is <https://www.monetdb.org/downloads/>. ### MonetDB Common - Fixed a regression where after a while the write-ahead log files weren't being rotated, meaning from some point onwards, the newest file just kept on growing. - When saving the SQL catalog during a low-level commit, we should only save the part of the catalog that corresponds to the part of the write-ahead log that has been processed. What we did was save more, which resulted in the catalog containing references to tables and columns whose disk presence is otherwise only in the write-ahead log. - A bug was fixed where the administration of which bats were in use was interpreted incorrectly during startup, causing problems later. One symptom that has been observed was failure to startup with a message that the catalog tables could not be loaded. ### SQL Frontend - Added a missing interface function sys.timestamp_to_str with a TIMESTAMP (as opposed to TIMESTAMP WITH TIME ZONE) argument. The missing interface caused error messages being produced when the function was called with a TIMESTAMP argument, although it did give the correct result. ### Bug Fixes - [7300](https://github.com/MonetDB/MonetDB/issues/7300): Implement missing standard SQL DATE and TIMESTAMP functions - [7324](https://github.com/MonetDB/MonetDB/issues/7324): string_distance('method',str1, str2) as a generic distance function - [7409](https://github.com/MonetDB/MonetDB/issues/7409): Numpy table returning UDFs with variadic arguments - [7410](https://github.com/MonetDB/MonetDB/issues/7410): SIGSEGV cause database corruption

1 0

New Dec2023 stable branch created
by Sjoerd Mullender 13 Nov '23

13 Nov '23

I have created a new stable branch for the next feature release. This branch was created in order to stabilize the code before the feature release. From now on *only* bug fixes may go to this new stable branch. I reserve the right to undo any changes that do not comply with this rule. I also reserve the right to admit changes that are not strictly bug fixes. The name for the branch is Dec2023. So in each of your stable checkout directories you can do hg pull hg update -rDec2023 to update to this new branch. -- Sjoerd Mullender

1 0

Announcement: New Jun2023-SP2 Bugfix release of MonetDB suite
by Sjoerd Mullender 03 Oct '23

03 Oct '23

The MonetDB team at MonetDB Solutions BV is pleased to announce the Jun2023-SP2 bugfix release of the MonetDB suite of programs. More information about MonetDB can be found on our website at <https://www.monetdb.org/>. For details on this release, please see the release notes at <https://www.monetdb.org/release-notes/jun2023/>. As usual, the download location is <https://www.monetdb.org/downloads/>. ### - Fixed an installation issue on Debian and Ubuntu introduced in the last build. - Do a lot more error checking, mostly for allocation failures. More is still needed, though. - Improve performance of the ILIKE operator when the pattern contains only ASCII characters. In this case we do not need to treat any characters as UTF-8 and we can use much faster routines that perform byte comparisons. ### MonetDB Common - Fixed a number of data races (race conditions). - Fixed a reference counting problem when a BAT could nog be loaded, e.g. because of resource limitations. - Only check for virtual memory limits when creating or growing bats, not for general memory allocations. There is (still) too much code that doesn't properly handle failing allocations, so we need to avoid those as much as possible. This has mostly an effect if there are virtual memory size restrictions imposed by cgroups (memory.swap.max in cgroups v2, memory.memsw.limit_in_bytes in cgroups v1). - The low-level commit turned out to always commit every persistent bat in the system. There is no need for that, it should only commit bats that were changed. This has now been fixed. - Implemented timeout/exit checks in a bunch more operators. Long(er) running operators occasionally check whether they're taking too long (past a user-specified timeout) or whether the server is exiting. This is now done in more places. ### Bug Fixes - [7094](https://github.com/MonetDB/MonetDB/issues/7094): Drop remote tables in transactions and rollback - [7303](https://github.com/MonetDB/MonetDB/issues/7303): Improve the performance of multi-column filters - [7400](https://github.com/MonetDB/MonetDB/issues/7400): VM max memory is not check correctly for cgroups v2 - [7401](https://github.com/MonetDB/MonetDB/issues/7401): Column aliases used incorrectly in UNION subqueries - [7402](https://github.com/MonetDB/MonetDB/issues/7402): Privileges on merge table not propagated to partition tables

1 0

Announcement: New Jun2023-SP1 Bugfix release of MonetDB suite
by Sjoerd Mullender 25 Jul '23

25 Jul '23

The MonetDB team at MonetDB Solutions BV is pleased to announce the Jun2023-SP1 bugfix release of the MonetDB suite of programs. More information about MonetDB can be found on our website at <https://www.monetdb.org/>. For details on this release, please see the release notes at <https://www.monetdb.org/release-notes/jun2023/>. As usual, the download location is <https://www.monetdb.org/downloads/>. ### MonetDB5 Server - There is now a new option --set tablet_threads=N to limit the number of threads used for a COPY INTO from CSV file query. This option can also be set for a specific database using the monetdb command using the ncopyintothreads property. ### SQL Frontend - An upgrade that both creates a new .snapshot user and extends the sys.db_user_info table with (among others) a password column did these in such a way that the passord value for the new user was NULL. This is fixed by updating the password. ### Bug Fixes - [7388](https://github.com/MonetDB/MonetDB/issues/7388): Query results in large cross product - [7394](https://github.com/MonetDB/MonetDB/issues/7394): BBPextend: ERROR: trying to extend BAT pool beyond the limit (163840000)

1 0

Announcement: New Jun2023 Feature release of MonetDB suite
by Panagiotis Koutsourakis 29 Jun '23

29 Jun '23

The MonetDB team at MonetDB Solutions BV is pleased to announce the Jun2023 feature release of the MonetDB suite of programs. More information about MonetDB can be found on our website at <https://www.monetdb.org/>. For details on this release, please see the release notes at <https://www.monetdb.org/release-notes/jun2023/>. As usual, the download location is <https://www.monetdb.org/downloads/>. The Jun2023 documentation can be found [here](https://www.monetdb.org/documentation-Jun2023/). ## Jun2023 Feature Release (11.47.3) ### Client Package - The COPY INTO from file ON CLIENT was extended to also look for a relative path name relative to the file from which the query was read. This is only possible if the name of the query file is known, so when it is specified on the command line or read using the interactive \< command. ### Mapi Library - Deprecate mapi_setfilecallback() in favor of mapi_setfilecallback2() which can handle binary downloads. For the time being, the old callback still works. ### ODBC Driver - Extended Windows MonetDB ODBC Data Source setup program with option to specify a logfile to enable tracing of ODBC Driver API calls. On other platforms users can edit the odbc.ini file and add a line: logfile=/home/username/odbctrace.log When a logfile is specified it will start logging the ODBC Driver API calls to the logfile after a new connection is made via SQLConnect() or SQLDriverConnect() or SQLBrowseConnect(). Note that enabling ODBC logging will slow down the performance of ODBC applications, so enable it only for analysing ODBC Driver problems. - Enhanced SQLTables() by adding support for table type names: 'BASE TABLE', 'GLOBAL TEMPORARY' and 'LOCAL TEMPORARY' in parameter TableType. These are synonyms of: 'TABLE', 'GLOBAL TEMPORARY TABLE' and 'LOCAL TEMPORARY TABLE'. ### MonetDB5 Server - The MAL debugger code has been removed. ### SQL Frontend - Add MAX_MEMORY and MAX_WORKERS options to the ALTER USER statement - Function 'similarity(x string, y string)' marked as deprecated and will be removed in the next release. - New functionality for string matching and similarity: startswith, endswith, contains, Levenshtein distance and Jaro-Winkler similarity. The functions startswith, endswith and contains have a version where a case insentive flag can be used. Also, there are new custom join functionality for startswith, endswith, contains, Levenshtein distance and Jaro-Winkler similarity. - Renamed previous Levenshtein distance to Damerau-Levenshtein distance. - New string function that transform from UTF-8 encoding to Ascii called asciify. - Session timeout feature improvement to start evaluating from the moment the procedure is called, instead of beginning of the session. - Queries stopped with the stop procedure are now marked as 'aborted' 'finished'. - The DEBUG statement has been removed. - SQL function sys.queue() overloaded with sys.queue(username string), SYSADMIN only, allowing to filter the global queue by username or use 'ALL' to retrieve the global queue. Calling the function without arguments returns the queue for the current user. - SQL procedures sys.pause(tag bigint), sys.resume(tag bigint), sys.stop(tag bigint) overloaded with sys.pause(tag bigint, username string), sys.resume(tag bigint, username string) and sys.stop(tag bigint, username string), SYSADMIN only, allowing to pause, resume and stop query executions by TAG, USERNAME. The call without arguments is a public procedure giving access to users to pause, resume and stop their own query executions. - Added support of ODBC escape sequences syntax to SQL layer. Now all clients (including ODBC/JDBC/pymonetdb) can use them without further processing. For details on ODBC escape sequences syntax see: https://learn.microsoft.com/en-us/sql/odbc/reference/appendixes/odbc-escape… - It is no longer allowed to create a merge table or remote table or replica table or unlogged table in schema "tmp". The tmp schema is reserved for temporary objects only, such as local/global temp tables. - System views sys.dependency_tables_on_functions and dependency_views_on_functions have been extended with column: function_id. - When loading data using COPY BINARY INTO, apply default values instead of just inserting NULLs. - When loading data using COPY BINARY INTO, validate DECIMAL(prec,scale) and VARCHAR(n) column width. - When loading data using COPY BINARY INTO, string used to have their line endings converted from CR LF to LF. Do not do this, it is the responsibility of the client. - Implemented dumping binary data using COPY SELECT ... INTO BINARY <file(s)>. - Removed code for Workload Capture and Replace, including system schemas "wlc" and "wlr" and the objects in those schemas. The code was experimental, and it didn't work out. A different approach will be taken. ### Merovingian - The monetdb get, inherit, lock and release commands are extended with ‘apply-to-all’ syntax similar to related functionality: Usage: monetdb set property=value [database ...] sets property to value for the given database(s), or all Usage: monetdb inherit property [database ...] unsets property, reverting to its inherited value from the default configuration for the given database(s), or all Usage: monetdb lock [-a] database [database...] Puts the given database in maintenance mode. Options: -a locks all known databases Usage: monetdb release [-a] database [database ...] Brings back a database from maintenance mode. Options: -a releases all known databases ### Bug Fixes - [7311](https://github.com/MonetDB/MonetDB/issues/7311): Missing `REGEXP_REPLACE` function. - [7344](https://github.com/MonetDB/MonetDB/issues/7344): Database upgrade failure due to user object dependency on system procedure - [7348](https://github.com/MonetDB/MonetDB/issues/7348): Subquery inside case always evaluated - [7378](https://github.com/MonetDB/MonetDB/issues/7378): MonetDB server crashes at sql_trans_copy_key - [7379](https://github.com/MonetDB/MonetDB/issues/7379): MonetDB server 11.46.0 crashes at cs_bind_ubat - [7380](https://github.com/MonetDB/MonetDB/issues/7380): MonetDB server 11.46.0 crashes at `BLOBcmp` - [7381](https://github.com/MonetDB/MonetDB/issues/7381): MonetDB server 11.46.0 crashes at `log_create_delta` - [7382](https://github.com/MonetDB/MonetDB/issues/7382): MonetDB server 11.46.0 crashes at `gc_col` - [7383](https://github.com/MonetDB/MonetDB/issues/7383): MonetDB server 11.46.0 crashes at `list_append` - [7384](https://github.com/MonetDB/MonetDB/issues/7384): MonetDB server 11.46.0 crashes at `__nss_database_lookup` - [7386](https://github.com/MonetDB/MonetDB/issues/7386): MonetDB server 11.46.0 crashes in `rel_deps` - [7387](https://github.com/MonetDB/MonetDB/issues/7387): MonetDB server 11.46.0 crashes in `rel_sequences`

1 0

Announcement: New Sep2022-SP3 Bugfix release of MonetDB suite
by Sjoerd Mullender 08 Jun '23

08 Jun '23

The MonetDB team at MonetDB Solutions BV is pleased to announce the Sep2022-SP3 bugfix release of the MonetDB suite of programs. More information about MonetDB can be found on our website at <https://www.monetdb.org/>. For details on this release, please see the release notes at <https://www.monetdb.org/release-notes/sep2022/>. As usual, the download location is <https://www.monetdb.org/downloads/>. ### Client Package - If the number of rows in mclient is set to 0 (using either --rows=0 option or \r0 on the mclient command line), the internal pager is used and it then uses the height of the terminal window. ### MonetDB Common - Warnings and informational messages are now sent to stdout instead of stderr, which means that monetdbd will now log them with the tag MSG instead of ERR. - Fixed parsing of the BBP.dir file when BAT ids grow larger than 2**24 (i.e. 100000000 in octal). - Fixed yet another occurrence of a missing .tailN file. This one could happen if a string bat was appended to in stages so that between appends the column was committed. If an append caused both a realloc of the tail heap because it was getting longer and a realloc because it got wider, the file might get removed before the GDK level commit happened. - When processing the WAL, if a to-be-destroyed object cannot be found, don't stop, but keep processing the rest of the WAL. - A race condition was fixed where certain write-ahead log messages could get intermingled, resulting in a corrupted WAL file. - If opening of a file failed when it was supposed to get memory mapped, an incorrect value was returned to indicate the failure, causing crashes later on. This has been fixed. - A race condition was fixed that could result in a missing tail file for a string bat (i.e. a file with .tail1, .tail2, or .tail4 extension). - When saving a bat failed for some reason during a low-level commit, this was logged in the log file, but the error was then subsequently ignored, possibly leading to files that are too short or even missing. - The write-ahead log (WAL) is now rotated a bit more efficiently by doing multiple log files in one go (i.e. in one low-level transaction). ### MonetDB5 Server - Client connections are cleaned up better so that we get fewer instances of clients that cannot connect. ### SQL Frontend - Fixed a regression introduced in the previous build: when a table is dropped in the same transaction where it was created, the data wasn't removed from disk. - When creating a hot snapshot, allow other clients to proceed, even with updating queries. - Increased the size of a variable counting the number of changes made to the database (e.g. in case more than 2 billion rows are added to a table). - Improved cleanup after failures such as failed memory allocations. - An insert into a table from which a column was dropped in a parallel transaction was incorrectly not flagged as a transaction conflict. - Fix bug where boolean NULLs were not recognized by COPY BINARY INTO ### Bug Fixes - [7360](https://github.com/MonetDB/MonetDB/issues/7360): Aggregates returning string crash - [7361](https://github.com/MonetDB/MonetDB/issues/7361): Table data is lost on DB restart, but only when a table has 2147483647 or more rows.

1 0

New Jun2023 stable branch created
by Sjoerd Mullender 19 Apr '23

19 Apr '23

I have created a new stable branch for the next feature release. This branch was created in order to stabilize the code before the feature release. From now on *only* bug fixes may go to this new stable branch. I reserve the right to undo any changes that do not comply with this rule. I also reserve the right to admit changes that are not strictly bug fixes. The name for the branch is Jun2023. So in each of your stable checkout directories you can do hg pull hg update -rJun2023 to update to this new branch. -- Sjoerd Mullender

1 0

Announcement: New MonetDB JDBC driver and JdbcClient released
by Martin van Dinther 02 Mar '23

02 Mar '23

The MonetDB team at MonetDB Solutions BV is pleased to announce a bug fix release of the MonetDB JDBC driver (monetdb-jdbc-3.3.jre8.jar) and JdbcClient CLI program (jdbcclient.jre8.jar). Besides bug fixes the new release improves performance and compliance. For a list of all changes see: https://www.monetdb.org/downloads/Java/ChangeLog We recommend all users of MonetDB JDBC driver and JdbcClient program to download and use the latest jar files. The download location is: https://www.monetdb.org/downloads/Java/ For information and tips on using the MonetDB JDBC driver, read: https://www.monetdb.org/downloads/Java/release.txt For information on using the MonetDB JDBC driver to execute COPY INTO <table> FROM <file> ON CLIENT ... and/or COPY <SELECT query> INTO <file> ON CLIENT ... commands for fast client side CSV file data import and export, read: https://www.monetdb.org/downloads/Java/onclient.txt Reminder: since release monetdb-jdbc-3.0.jre8.jar the MonetDB JDBC Driver class name has been changed to: org.monetdb.jdbc.MonetDriver It used to be nl.cwi.monetdb.jdbc.MonetDriver which is marked as deprecated and will be removed in a future release. Please use the new driver class name asap in your configuration files/scripts and/or Java code.

1 0

Announcement: New Sep2022-SP2 Bugfix release of MonetDB suite
by Sjoerd Mullender 31 Jan '23

31 Jan '23

The MonetDB team at MonetDB Solutions BV is pleased to announce the Sep2022-SP2 bugfix release of the MonetDB suite of programs. More information about MonetDB can be found on our website at <https://www.monetdb.org/>. For details on this release, please see the release notes at <https://www.monetdb.org/release-notes/sep2022/>. As usual, the download location is <https://www.monetdb.org/downloads/>. ### ODBC Driver - A crash in the ODBC driver was fixed when certain unsupported functions where used in a {fn ...} escape. - Prepare of a query where the sum of the number of parameters (question marks in the query) and the number of output columns is larger than 100 could fail with an unexpected error. This has been fixed. ### MonetDB Common - Fixed a race condition that could lead to a bat being added to the SQL catalog but nog being made persistent, causing a subsequent restart of the system to fail (and crash). - Fixed a race condition where a hash could have been created on a bat using the old bat count while in another thread the bat count got updated. This would make the hash be based on too small a size, causing failures later on. - When extending a bat failed, the capacity had been updated already and was therefore too large. This could then later cause a crash. This has been fixed by only updating the capacity if the extend succeeded. ### SQL Frontend - Fixed a regression where when there are multiple concurrent transactions, the dependencies weren't checked properly. - Added some error checking to prevent crashes. Errors would mainly occur under memory pressure. - Fixed cleanup after a failed allocation where the data being cleaned up was unitialized but still used as pointers to memory that also had to be freed. - Fixed a double cleanup after a failed allocation in COPY INTO. The double cleanup could cause a crash due to a race condition it enabled. ### Bug Fixes - [7343](https://github.com/MonetDB/MonetDB/issues/7343): GDKmmap requesting 0 virtual memory - [7347](https://github.com/MonetDB/MonetDB/issues/7347): A bug where an exception occurs even though it is a query with normal syntax (Merge Table)

1 0

Announcement: New Sep2022-SP1 Bugfix release of MonetDB suite
by Sjoerd Mullender 09 Dec '22

09 Dec '22

The MonetDB team at MonetDB Solutions BV is pleased to announce the Sep2022-SP1 bugfix release of the MonetDB suite of programs. More information about MonetDB can be found on our website at <https://www.monetdb.org/>. For details on this release, please see the release notes at <https://www.monetdb.org/release-notes/sep2022/>. As usual, the download location is <https://www.monetdb.org/downloads/>. ### Client Package - Also dump the new options of CREATE USER. ### ODBC Driver - Fixed issue with generated raw strings prefix when ODBC driver is used against a server older than Jun2020 (11.37). ### MonetDB Common - On Windows, use the wide-character interface of system calls when dealing with the environment (i.e. file names and getenv()). - Memory leaks have been fixed. - Improved maintenance of the estimated number of distinct values in BATs. The estimate helps in deciding which low-level algorithm to use. - Offset heaps (.tailN files) were growing too fast and unnecessarily under certain conditions. This has been fixed. Also, when such too large files are now loaded into the system, it is recognized they are too large and they are truncated to a more reasonable size. ### MonetDB5 Server - Fixed a crash when the server runs out of client contexts (i.e. more concurrent clients than the server is configured to handle). - A race condition in the SHA hash code was fixed which resulted in occasional failed connection attempts when they occurred concurrently. - Fix a bug where the MAL optimizer would use the starttime of the previous query to determine whether a query timeout occurred. ### SQL Frontend - Improved the handling of the "idle" value in the sys.sessions function and view. ### Merovingian - Stop logging references to monetdbd's logfile in said logfile. ### Bug Fixes - [7330](https://github.com/MonetDB/MonetDB/issues/7330): Creating temporary table fails after reconnect - [7333](https://github.com/MonetDB/MonetDB/issues/7333): DLLs fail to load on Windows with accented characters in path - [7336](https://github.com/MonetDB/MonetDB/issues/7336): Selecting from a literal-value table returns wrong values - [7339](https://github.com/MonetDB/MonetDB/issues/7339): MonetDB corrupted state after SIGKILL - [7342](https://github.com/MonetDB/MonetDB/issues/7342): column which datatype is double couldn't group or aggregation in version 11.45.7

1 0

Announcement: New Sep2022 Feature release of MonetDB suite
by Sjoerd Mullender 26 Sep '22

26 Sep '22

The MonetDB team at MonetDB Solutions BV is pleased to announce the Sep2022 feature release of the MonetDB suite of programs. More information about MonetDB can be found on our website at <https://www.monetdb.org/>. For details on this release, please see the release notes at <https://www.monetdb.org/documentation/admin-guide/release-notes/sep2022/>. As usual, the download location is <https://www.monetdb.org/downloads/>. The Sep2022 documentation can be found [here](https://www.monetdb.org/documentation-Sep2022/). ## Sep2022 Feature Release (11.45.7) ### Client Package - Dumping of function GRANTs was improved by adding the types of the function (and procedure) arguments. ### ODBC Driver - Improved SQLSetPos(). It now allows RowNumber to be 0 when Operation is SQL_POSITION. - Improved ODBC Error/Warning messages. They now include the name of the Data Source as required by the ODBC specification: [MonetDB][ODBC driver VERSION][data-source-name] data-source-supplied-text - Improved MonetDB ODBC Data Source Configuration dialog on MS Windows by hiding the typed in password text. It now shows dots for the characters. This fixes request https://github.com/MonetDB/MonetDB/issues/7314 - Changed output of TABLE_CAT or PROCEDURE_CAT result columns as returned by ODBC functions: SQLTables(), SQLColumns(), SQLPrimaryKeys(), SQLForeignKeys(), SQLStatistics(), SQLTablePrivileges(), SQLColumnPrivileges(), SQLProcedures() and SQLProcedureColumns(). They used to return the static database name but now they will return NULL as MonetDB does not support CATALOG objects or qualifiers. - Removed the possibility to retrieve or set the CURRENT_CATALOG via SQLGetConnectAttr(hdbc, SQL_ATTR_CURRENT_CATALOG, ...) and SQLSetConnectAttr(hdbc, SQL_ATTR_CURRENT_CATALOG, ...) as MonetDB does not support CATALOG objects (no SQL support for: CREATE CATALOG abc or SET CATALOG abc) and therefore there is no CURRENT_CATALOG. - Corrected ODBC functions SQLTablePrivileges() and SQLColumnPrivileges() for local temporary tables located in schema tmp. They did not return any rows when the temporary table had privileges set. Now they do return rows as expected. - Improved SQLProcedures() and SQLProcedureColumns(). They now list information also for all built-in system procedures and functions, not only those created via SQL. Also corrected the value of ORDINAL_POSITION for scalar function arguments. It would start at 2 instead of 1. - Extended output of SQLProcedures() and SQLProcedureColumns() resultsets with an extra column SPECIFIC_NAME. This column contains the name which uniquely identifies this procedure or function within its schema. As MonetDB supports overloaded procedures and functions, the combination of PROCEDURE_SCHEM and PROCEDURE_NAME is not enough to uniquely identify a procedure or function. This extra column allows you to correctly match the corresponding rows returned by SQLProcedureColumns() with the specific rows of SQLProcedures(). This extra column SPECIFIC_NAME is implemented similar to the JDBC DatabaseMetaData methods getProcedures() and getProcedureColumns(). - For SQLForeignKeys() corrected the output of columns UPDATE_RULE and DELETE_RULE. These columns used to always return 3 (= SQL_NO_ACTION) but now they will report the action codes as specified in the FOREIGN KEY CONSTRAINT construction. - Fixed issue in function SQLProcedureColumns(). When the argument ColumnName was not empty it generated an invalid SQL query which failed with error: SELECT: no such column 'c.name'. This has been resolved. - Corrected implementation of SQLGetInfo(hdbc, SQL_MAX_DRIVER_CONNECTIONS, ...). It used to always return 64. Now it returns the value from the MonetDB server. - Changed the column names case of the result sets as returned by SQLTables(), SQLColumns(), SQLSpecialColumns(), SQLPrimaryKeys(), SQLForeignKeys(), SQLStatistics(), SQLTablePrivileges(), SQLColumnPrivileges(), SQLProcedures() and SQLProcedureColumns(). The column names where all in lowercase but the ODBC specification defines them in uppercase, so changed them to uppercase. - Corrected SQLSpecialColumns(..., SQL_BEST_ROWID, ...). Previously it only returned rows when the table had a primary key. Now it also returns rows when a table has a unique constraint. - Corrected SQLStatistics(..., SQL_INDEX_ALL, ...). Previously it only returned rows when the table had a primary or unique key. Now it also returns rows for indices which are not based on a key. - Corrected SQLStatistics(..., SQL_ENSURE). It now returns CARDINALITY information for columns based on a primary/unique key. Previously it always returned NULL for the CARDINALITY result column. - Corrected ODBC functions SQLPrimaryKeys(), SQLSpecialColumns() and SQLStatistics() for local temporary tables located in schema tmp. They did not return any rows when the temp table had a primary or unique key or index. Now they do return rows as expected. ### MonetDB Common - Fixed a bug in ORDER BY with both NULLS LAST and LIMIT when the ordering was on an interger or integer-like column and NULL values are present. - The median_avg and quantile_avg returned bogus results in the non-grouped case (i.e. something like SELECT sys.median_avg(i) FROM t). - The abort_on_error parameter of all GDK-level functions has been removed. Errors (e.g. overflow) now also results in an error. - Implemented BC/AD (and BCE/CE) suffixes when parsing dates. - The interface for using strimps has not changed (create an imprint index on a column of a read only table), but now construction happens at the beginning of the first query that uses the strimp and is performed in a multithreaded manner. - Implemented the use of strimps for NOT LIKE queries. The idea is to run the corresponding LIKE query using strimps and take the complement of the result. We keep around NULL values both during strimp filtering and during the pcre part of the LIKE query so that they get discarded automatically when we take the complement. - The function BBPkeepref now gets a BAT pointer as argument instead of a bat id. - Get rid of macro Tsize, use ->twidth instead. - Get rid of macro BUNlast, just use BATcount instead. - The BLOB type has been moved into the GDK layer. - When adding or subtracting months from a date or timestamp value, clamp the result to the calculated month instead of wrapping to the beginning of the next month. See bug [7227](https://https://github.com/MonetDB/MonetDB/issues/7227). - The median_avg and quantile_avg returned bogus results in the non-grouped case (i.e. something like SELECT sys.median_avg(i) FROM t). ### MonetDB5 Server - Disabled volcano pipeline due to known issues. ### SQL Frontend - The function sys.tracelog is now executable by anyone. The function (and view of the same name) returns the tracing information of a query that was prepended with the TRACE keyword. - Fixed a bug in COPY BINARY INTO where the input wasn't checked thoroughly enough. - Extended the built-in profiler to emit non-MAL events related to query compilation, optimization, transactions and client connections. To minimize, simplify and optimize the process of generating and processing profiler output, only the events marking the end of an operation are emitted in most cases and the emitted json messages themselves are trimmed down to their essential fields. Furthermore, the MAL instruction profiler.openstream now requires an integer as a single argument, "0" for default behaviour or "4" to turn on the profiler in minimal mode, which causes it to only emit general events and excludes individual MAL instruction execution events from the profiler streams. The MAL instruction profiler.openstream with zero arguments is deprecated. - Extended system view sys.users with columns: schema_path, max_memory, max_workers, optimizer and default_role. Extended system table sys.db_user_info with columns: schema_path, max_memory, max_workers, optimizer, default_role and password. The password is encrypted. This table can only be queried when the user has been granted the select privilege from monetdb. - CREATE USER statement has been extended to take more optional arguments like MAX_MEMORY, MAX_WORKERS, OPTIMIZER, DEFAULT ROLE. ALTER USER statement has also been extended with DEFAULT ROLE. - A new function sys.url_extract_host(string, bool) which returns the host name from the given URL has been implemented. The bool argument, if true, causes the www. prefix to be removed. - The user authentication tables are now administered by the SQL layer instead of the MAL layer. This means that changing (adding, dropping, altering) user and role information is now transaction-safe. - Added the UNLOGGED TABLE feature. - Removed functions sys.index and sys.strings. ### Merovingian - Added "loglevel" property for monetdbd logging (to merovingian.log). The loglevel can be set to: error or warning or information or debug. The loglevel property can be changed dynamically via command: monetdbd set loglevel=warning /path/to/dbfarm Default the loglevel is set to: information When loglevel is error, only errors are logged. When loglevel is warning, errors and warnings are logged. When loglevel is information, errors and warnings and information messages are logged. When loglevel is debug, all messages are logged. - There is now a loadmodules property that can be used to add --loadmodule arguments to the mserver5 command line. See the monetdb and mserver5 manual pages. - In certain cases (when an mserver5 process exits right after producing a message) the log message was logged over and over again, causing monetdbd to use 100% CPU. This has been fixed. ### Bug Fixes - [6982](https://github.com/MonetDB/MonetDB/issues/6982): Wrong constraint name in error message of constraint violation - [7209](https://github.com/MonetDB/MonetDB/issues/7209): Configuration option for merovingian.log - [7225](https://github.com/MonetDB/MonetDB/issues/7225): Invalid memory access when extending a BAT during appends - [7227](https://github.com/MonetDB/MonetDB/issues/7227): Date calculations, bug or feature - [7273](https://github.com/MonetDB/MonetDB/issues/7273): Concurrent reads and writes causes "BATproject2: does not match always" error - [7282](https://github.com/MonetDB/MonetDB/issues/7282): call sys.dump_table_data(); fails - [7285](https://github.com/MonetDB/MonetDB/issues/7285): C-UDFs: aggr_group.count has wrong value (number of input rows instead of number of groups). - [7296](https://github.com/MonetDB/MonetDB/issues/7296): Implictly cast a timestamp string to DATE when appropriate - [7297](https://github.com/MonetDB/MonetDB/issues/7297): Parsing partial dates behaves unpredictable - [7306](https://github.com/MonetDB/MonetDB/issues/7306): ODBC Driver Assertion `stmt->Dbc->FirstStmt' Failed - [7314](https://github.com/MonetDB/MonetDB/issues/7314): ODBC Driver : please mask/hide password - [7318](https://github.com/MonetDB/MonetDB/issues/7318): distinct in a subquery not working properly

1 0

Announcement: New Sep2022 Feature release of MonetDB suite
by Sjoerd Mullender 26 Sep '22

26 Sep '22

The MonetDB team at MonetDB Solutions BV is pleased to announce the Sep2022 feature release of the MonetDB suite of programs. More information about MonetDB can be found on our website at <https://www.monetdb.org/>. For details on this release, please see the release notes at <https://www.monetdb.org/documentation/admin-guide/release-notes/sep2022/>. As usual, the download location is <https://www.monetdb.org/downloads/>. The Sep2022 documentation can be found [here](https://www.monetdb.org/documentation-Sep2022/). ## Sep2022 Feature Release (11.45.7) ### Client Package - Dumping of function GRANTs was improved by adding the types of the function (and procedure) arguments. ### ODBC Driver - Improved SQLSetPos(). It now allows RowNumber to be 0 when Operation is SQL_POSITION. - Improved ODBC Error/Warning messages. They now include the name of the Data Source as required by the ODBC specification: [MonetDB][ODBC driver VERSION][data-source-name] data-source-supplied-text - Improved MonetDB ODBC Data Source Configuration dialog on MS Windows by hiding the typed in password text. It now shows dots for the characters. This fixes request https://github.com/MonetDB/MonetDB/issues/7314 - Changed output of TABLE_CAT or PROCEDURE_CAT result columns as returned by ODBC functions: SQLTables(), SQLColumns(), SQLPrimaryKeys(), SQLForeignKeys(), SQLStatistics(), SQLTablePrivileges(), SQLColumnPrivileges(), SQLProcedures() and SQLProcedureColumns(). They used to return the static database name but now they will return NULL as MonetDB does not support CATALOG objects or qualifiers. - Removed the possibility to retrieve or set the CURRENT_CATALOG via SQLGetConnectAttr(hdbc, SQL_ATTR_CURRENT_CATALOG, ...) and SQLSetConnectAttr(hdbc, SQL_ATTR_CURRENT_CATALOG, ...) as MonetDB does not support CATALOG objects (no SQL support for: CREATE CATALOG abc or SET CATALOG abc) and therefore there is no CURRENT_CATALOG. - Corrected ODBC functions SQLTablePrivileges() and SQLColumnPrivileges() for local temporary tables located in schema tmp. They did not return any rows when the temporary table had privileges set. Now they do return rows as expected. - Improved SQLProcedures() and SQLProcedureColumns(). They now list information also for all built-in system procedures and functions, not only those created via SQL. Also corrected the value of ORDINAL_POSITION for scalar function arguments. It would start at 2 instead of 1. - Extended output of SQLProcedures() and SQLProcedureColumns() resultsets with an extra column SPECIFIC_NAME. This column contains the name which uniquely identifies this procedure or function within its schema. As MonetDB supports overloaded procedures and functions, the combination of PROCEDURE_SCHEM and PROCEDURE_NAME is not enough to uniquely identify a procedure or function. This extra column allows you to correctly match the corresponding rows returned by SQLProcedureColumns() with the specific rows of SQLProcedures(). This extra column SPECIFIC_NAME is implemented similar to the JDBC DatabaseMetaData methods getProcedures() and getProcedureColumns(). - For SQLForeignKeys() corrected the output of columns UPDATE_RULE and DELETE_RULE. These columns used to always return 3 (= SQL_NO_ACTION) but now they will report the action codes as specified in the FOREIGN KEY CONSTRAINT construction. - Fixed issue in function SQLProcedureColumns(). When the argument ColumnName was not empty it generated an invalid SQL query which failed with error: SELECT: no such column 'c.name'. This has been resolved. - Corrected implementation of SQLGetInfo(hdbc, SQL_MAX_DRIVER_CONNECTIONS, ...). It used to always return 64. Now it returns the value from the MonetDB server. - Changed the column names case of the result sets as returned by SQLTables(), SQLColumns(), SQLSpecialColumns(), SQLPrimaryKeys(), SQLForeignKeys(), SQLStatistics(), SQLTablePrivileges(), SQLColumnPrivileges(), SQLProcedures() and SQLProcedureColumns(). The column names where all in lowercase but the ODBC specification defines them in uppercase, so changed them to uppercase. - Corrected SQLSpecialColumns(..., SQL_BEST_ROWID, ...). Previously it only returned rows when the table had a primary key. Now it also returns rows when a table has a unique constraint. - Corrected SQLStatistics(..., SQL_INDEX_ALL, ...). Previously it only returned rows when the table had a primary or unique key. Now it also returns rows for indices which are not based on a key. - Corrected SQLStatistics(..., SQL_ENSURE). It now returns CARDINALITY information for columns based on a primary/unique key. Previously it always returned NULL for the CARDINALITY result column. - Corrected ODBC functions SQLPrimaryKeys(), SQLSpecialColumns() and SQLStatistics() for local temporary tables located in schema tmp. They did not return any rows when the temp table had a primary or unique key or index. Now they do return rows as expected. ### MonetDB Common - Fixed a bug in ORDER BY with both NULLS LAST and LIMIT when the ordering was on an interger or integer-like column and NULL values are present. - The median_avg and quantile_avg returned bogus results in the non-grouped case (i.e. something like SELECT sys.median_avg(i) FROM t). - The abort_on_error parameter of all GDK-level functions has been removed. Errors (e.g. overflow) now also results in an error. - Implemented BC/AD (and BCE/CE) suffixes when parsing dates. - The interface for using strimps has not changed (create an imprint index on a column of a read only table), but now construction happens at the beginning of the first query that uses the strimp and is performed in a multithreaded manner. - Implemented the use of strimps for NOT LIKE queries. The idea is to run the corresponding LIKE query using strimps and take the complement of the result. We keep around NULL values both during strimp filtering and during the pcre part of the LIKE query so that they get discarded automatically when we take the complement. - The function BBPkeepref now gets a BAT pointer as argument instead of a bat id. - Get rid of macro Tsize, use ->twidth instead. - Get rid of macro BUNlast, just use BATcount instead. - The BLOB type has been moved into the GDK layer. - When adding or subtracting months from a date or timestamp value, clamp the result to the calculated month instead of wrapping to the beginning of the next month. See bug [7227](https://https://github.com/MonetDB/MonetDB/issues/7227). - The median_avg and quantile_avg returned bogus results in the non-grouped case (i.e. something like SELECT sys.median_avg(i) FROM t). ### MonetDB5 Server - Disabled volcano pipeline due to known issues. ### SQL Frontend - The function sys.tracelog is now executable by anyone. The function (and view of the same name) returns the tracing information of a query that was prepended with the TRACE keyword. - Fixed a bug in COPY BINARY INTO where the input wasn't checked thoroughly enough. - Extended the built-in profiler to emit non-MAL events related to query compilation, optimization, transactions and client connections. To minimize, simplify and optimize the process of generating and processing profiler output, only the events marking the end of an operation are emitted in most cases and the emitted json messages themselves are trimmed down to their essential fields. Furthermore, the MAL instruction profiler.openstream now requires an integer as a single argument, "0" for default behaviour or "4" to turn on the profiler in minimal mode, which causes it to only emit general events and excludes individual MAL instruction execution events from the profiler streams. The MAL instruction profiler.openstream with zero arguments is deprecated. - Extended system view sys.users with columns: schema_path, max_memory, max_workers, optimizer and default_role. Extended system table sys.db_user_info with columns: schema_path, max_memory, max_workers, optimizer, default_role and password. The password is encrypted. This table can only be queried when the user has been granted the select privilege from monetdb. - CREATE USER statement has been extended to take more optional arguments like MAX_MEMORY, MAX_WORKERS, OPTIMIZER, DEFAULT ROLE. ALTER USER statement has also been extended with DEFAULT ROLE. - A new function sys.url_extract_host(string, bool) which returns the host name from the given URL has been implemented. The bool argument, if true, causes the www. prefix to be removed. - The user authentication tables are now administered by the SQL layer instead of the MAL layer. This means that changing (adding, dropping, altering) user and role information is now transaction-safe. - Added the UNLOGGED TABLE feature. - Removed functions sys.index and sys.strings. ### Merovingian - Added "loglevel" property for monetdbd logging (to merovingian.log). The loglevel can be set to: error or warning or information or debug. The loglevel property can be changed dynamically via command: monetdbd set loglevel=warning /path/to/dbfarm Default the loglevel is set to: information When loglevel is error, only errors are logged. When loglevel is warning, errors and warnings are logged. When loglevel is information, errors and warnings and information messages are logged. When loglevel is debug, all messages are logged. - There is now a loadmodules property that can be used to add --loadmodule arguments to the mserver5 command line. See the monetdb and mserver5 manual pages. - In certain cases (when an mserver5 process exits right after producing a message) the log message was logged over and over again, causing monetdbd to use 100% CPU. This has been fixed. ### Bug Fixes - [6982](https://github.com/MonetDB/MonetDB/issues/6982): Wrong constraint name in error message of constraint violation - [7209](https://github.com/MonetDB/MonetDB/issues/7209): Configuration option for merovingian.log - [7225](https://github.com/MonetDB/MonetDB/issues/7225): Invalid memory access when extending a BAT during appends - [7227](https://github.com/MonetDB/MonetDB/issues/7227): Date calculations, bug or feature - [7273](https://github.com/MonetDB/MonetDB/issues/7273): Concurrent reads and writes causes "BATproject2: does not match always" error - [7282](https://github.com/MonetDB/MonetDB/issues/7282): call sys.dump_table_data(); fails - [7285](https://github.com/MonetDB/MonetDB/issues/7285): C-UDFs: aggr_group.count has wrong value (number of input rows instead of number of groups). - [7296](https://github.com/MonetDB/MonetDB/issues/7296): Implictly cast a timestamp string to DATE when appropriate - [7297](https://github.com/MonetDB/MonetDB/issues/7297): Parsing partial dates behaves unpredictable - [7306](https://github.com/MonetDB/MonetDB/issues/7306): ODBC Driver Assertion `stmt->Dbc->FirstStmt' Failed - [7314](https://github.com/MonetDB/MonetDB/issues/7314): ODBC Driver : please mask/hide password - [7318](https://github.com/MonetDB/MonetDB/issues/7318): distinct in a subquery not working properly

1 0

Announcement: New Jan2022-SP4 Bugfix release of MonetDB suite
by Sjoerd Mullender 22 Aug '22

22 Aug '22

The MonetDB team at MonetDB Solutions BV is pleased to announce the Jan2022-SP4 bugfix release of the MonetDB suite of programs. More information about MonetDB can be found on our website at <https://www.monetdb.org/>. For details on this release, please see the release notes at <https://www.monetdb.org/documentation/admin-guide/release-notes/jan2022/>. As usual, the download location is <https://www.monetdb.org/downloads/>. ### Client Package - Implemented dump of global grants, that is to say, grants for COPY INTO and COPY FROM which grant permission to users to write to or read from files on the server (COPY INTO queries without the ON CLIENT option). - Fixed a bug where when the semicolon at the end of a COPY INTO query that reads from STDIN is at exactly a 10240 byte boundary in a file, the data isn't read as input for the COPY INTO but instead as a new SQL query. ### MonetDB Common - A bug was fixed when upgrading a database from the Oct2020 releases (11.39.X) or older when the write-ahead log (WAL) was not empty and contained instructions to create new tables. - When destroying a bat, make sure there are no files left over in the BACKUP directory since they can cause problems when the bat id gets reused. - Fixed an off-by-one error in the logger which caused older log files to stick around longer in the write-ahead log than necessary. - When an empty BAT is committed, skip writing (and synchronizing to disk) the heap (tail and theap) files and write 0 for their sizes to the BBP.dir file. When reading the BBP.dir file, if an empty BAT is encountered, set the sizes of those files to 0. This fixes potential issues during startup of the server (BBPcheckbats reporting errors). - Make sure heap files of transient bats get deleted when the bat is destroyed. If the bat was a partial view (sharing the vheap but not the tail), the tail file wasn't deleted. - Various changes were made to satisfy newer compilers. - The batDirtydesc and batDirtyflushed Boolean values have been deprecated and are no longer used. They were both holdovers from long ago. - Various race conditions (data races) have been fixed. - All accesses to the BACKUP directory need to be protected by the same lock. The lock already existed (GDKtmLock), but wasn't used consistently. This is now fixed. Hopefully this makes the hot snapshot code more reliable. ### MonetDB5 Server - Various race conditions (data races) have been fixed. ### Merovingian - When multiple identical messages are written to the log, write the first one, and combine subsequent ones in a single message. - Fixed a leak where the log file wasn't closed when it was reopened after a log rotation (SIGHUP signal). - Try to deal more gracefully with "inherited" mserver5 processes. This includes not complaining about an "impossible state", and allowing such processes to be stopped by the monetdbd process. - When a transient failure occurs during processing of a new connection to the monetdbd server, sleep for half a second so that if the transient failure occurs again, the log file doesn't get swamped with error messages. ### Bug Fixes - [7040](https://github.com/MonetDB/MonetDB/issues/7040): Memory leak detected for MAPI interface - [7298](https://github.com/MonetDB/MonetDB/issues/7298): Irresponsive database server after reading incomplete SQL script. - [7308](https://github.com/MonetDB/MonetDB/issues/7308): Race condition in MVCC transaction management

1 0

New Sep2022 stable branch created
by Sjoerd Mullender 10 Aug '22

10 Aug '22

I have created a new stable branch for the next feature release. This branch was created in order to stabilize the code before the feature release. From now on *only* bug fixes may go to this new stable branch. I reserve the right to undo any changes that do not comply with this rule. I also reserve the right to admit changes that are not strictly bug fixes. The name for the branch is Sep2022. So in each of your stable checkout directories you can do hg pull hg update -rSep2022 to update to this new branch. -- Sjoerd Mullender

1 0

Announcement: New Jan2022-SP3 Bugfix release of MonetDB suite
by Sjoerd Mullender 31 May '22

31 May '22

The MonetDB team at MonetDB Solutions BV is pleased to announce the Jan2022-SP3 bugfix release of the MonetDB suite of programs. More information about MonetDB can be found on our website at <https://www.monetdb.org/>. For details on this release, please see the release notes at <https://www.monetdb.org/documentation/admin-guide/release-notes/jan2022/>. As usual, the download location is <https://www.monetdb.org/downloads/>. ### MonetDB Common - All accesses to the BACKUP directory need to be protected by the same lock. The lock already existed (GDKtmLock), but wasn't used consistently. This is now fixed. Hopefully this makes the hot snapshot code more reliable. - When exiting, long running instructions are aborted using the same mechanism that is used for query timeouts. ### SQL Frontend - GLOBAL TEMPORARY tables are now treated like LOCAL TEMPORARY tables as far as the table content is concerned. The schema information stays global. This fixes an issue with concurrent access and cleanup of stale data. - The NO CONSTRAINT option of the COPY INTO query has been removed. It didn't work and it was never a good idea anyway. ### Bug Fixes - [7036](https://github.com/MonetDB/MonetDB/issues/7036): Generate column names instead of labels

1 0

Announcement: New Jan2022-SP2 Bugfix release of MonetDB suite
by Sjoerd Mullender 08 Apr '22

08 Apr '22

The MonetDB team at MonetDB Solutions BV is pleased to announce the Jan2022-SP2 bugfix release of the MonetDB suite of programs. More information about MonetDB can be found on our website at <https://www.monetdb.org/>. For details on this release, please see the release notes at <https://www.monetdb.org/documentation/admin-guide/release-notes/jan2022/>. As usual, the download location is <https://www.monetdb.org/downloads/>. ### Client Package - Improved the handling of the \r (internal pager) command in mclient. It now properly counts the header of table, and when a (very) long table is being printed and aborted part way in the built-in pager, not all data is transferred to the client (and then discarded). Instead at most 1000 rows are transferred. ### MonetDB Common - Improved speed of BATappend to empty varsized bat: we now just copy the heaps instead of inserting individual values. - Improved speed of projection (BATproject) on varsized bats by sharing the data heap (vheap). - Fixed a race condition which could cause a too large size being written for a .theap file to the BBP.dir file after the correct size file had been saved to disk. - We now ignore the size and capacity columns in the BBP.dir file. These values are essential during run time, but not useful in the on-disk image of the database. - Fixed a bug in the append code for msk (bit mask) bats. - Conversions from floating point types to integral types that involve multiplication now use the "long double" as intermediate type, thereby loosing as few significant bits as is feasible. - Found and fixed another source for the now infamous BBPcheckbats error that sometimes occurs at startup of the server. ### MonetDB5 Server - Improved parsing speed of blob values, especially on Windows. On Windows, using the locale-aware functions isdigit and isxdigit is comparatively very slow, so we avoid them. ### Bug Fixes - [7252](https://github.com/MonetDB/MonetDB/issues/7252): Segmentation fault on second run - [7253](https://github.com/MonetDB/MonetDB/issues/7253): Extremely slow INSERT INTO <table> SELECT - [7254](https://github.com/MonetDB/MonetDB/issues/7254): Commit with deletions is very slow - [7263](https://github.com/MonetDB/MonetDB/issues/7263): PRIMARY KEY constraint is not persistent through server restarts - [7267](https://github.com/MonetDB/MonetDB/issues/7267): Update after delete does not update some rows - [7278](https://github.com/MonetDB/MonetDB/issues/7278): BUG when there is more than one field/filter in the having clause

1 0

Announcement: New Jan2022-SP1 Bugfix release of MonetDB suite
by Sjoerd Mullender 14 Feb '22

14 Feb '22

The MonetDB team at MonetDB Solutions BV is pleased to announce the Jan2022-SP1 bugfix release of the MonetDB suite of programs. More information about MonetDB can be found on our website at <https://www.monetdb.org/>. For details on this release, please see the release notes at <https://www.monetdb.org/documentation/admin-guide/release-notes/jan2022/>. As usual, the download location is <https://www.monetdb.org/downloads/>. ### SQL Frontend - [This feature was already released in Jan2022 (11.43), but the ChangeLog was missing] Added SQL procedures sys.vacuum(sname string, tname string, cname string), sys.vacuum(sname string, tname string, cname string, interval int), sys.stop_vacuum(sname string, tname string, cname string). These can be used to vacuum string columns. ### Bug Fixes - [7228](https://github.com/MonetDB/MonetDB/issues/7228): COMMIT: transaction is aborted because of concurrency conflicts, will ROLLBACK instead - [7230](https://github.com/MonetDB/MonetDB/issues/7230): Prepared statement of INSERT with SELECT fails when types difer - [7232](https://github.com/MonetDB/MonetDB/issues/7232): False conflicts when inserting in a not null field - [7237](https://github.com/MonetDB/MonetDB/issues/7237): SELECT with concurrent writes rarely returns corrupt data - [7238](https://github.com/MonetDB/MonetDB/issues/7238): query with system function: "index"(varchar, boolean) fails with GDK error or assertion failure. - [7241](https://github.com/MonetDB/MonetDB/issues/7241): Replacing a view by a query on the view itself crashes the server.

1 0

mailing list software updated
by Sjoerd Mullender 10 Feb '22

10 Feb '22

I have updated the mailing list software on monetdb.org from mailman2 to mailman3. This means that the old accounts for managing list subscriptions do not work anymore. In order to manage your subscription using a web interface, you will need to create a new account at https://www.monetdb.org/lists/. You can associate multiple email addresses to this account and manage all your subscriptions at monetdb.org using a single account, even if they use different email addresses. Unlike in mailman2, in mailman3 the passwords are not saved in plaintext, and therefore they cannot be sent as a monthly reminder. Mailing list archives have been converted to the new system as well. The old web addresses (https://mail.monetdb.org/, https://www.monetdb.org/mailman/ and https://www.monetdb.org/pipermail/) now all get redirected to the new interface. -- Sjoerd Mullender

1 0

None
by None 10 Feb '22

10 Feb '22

does, however, like M4 still have the strange behaviour to be slower on PreparedStatements. When swithing from server side prepared statements, to Java emulated prepared statements, the performance goes to the same level as simple batched statements, while Java wise, the emulated version is much more computational extensive. I'm under the impression that for update queries we have around the absolute minimum of data sent over at this moment, hence the 'gap' we have with MySQL must be in the engine and/or the driver. For the latter I will have to investigate where shortcuts can be made, or inefficient structures are being used (such as very frequent Map lookups with few entries). This benchmark only tests update queries (inserts, updates, deletes), hence there is a need to revive a select based benchmark, or create one to get an idea how we perform in other use cases. A conclusion from the latest improvements is that searching for newlines appears to be expensive in both Java as well as C. (Smack tests showed a 10x speedup). Future improvements on these responses from the server that are a tabular result can be: - change header format for easier parsing (I don't expect much from this) - replace newlines by integer lengths to avoid string scanning - make parsing of tabular results easier/less expensive by avoiding string escaping

1 0

New MonetDB JDBC driver v3.2 and JdbcClient released
by Martin van Dinther 03 Feb '22

03 Feb '22

The MonetDB team at MonetDB BV is pleased to announce a feature release of the MonetDB JDBC driver (monetdb-jdbc-3.2.jre8.jar) and JdbcClient CLI program (jdbcclient.jre8.jar). The new release adds the capability to use the SQL: COPY ... INTO ... ON CLIENT statements via Java from the client side. This allows users and programmers to use the much faster COPY statements to load large data sets into MonetDB from a local client side CSV file or stream. The JDBC MonetConnection API has been extended with methods to register and use Upload and Download Handlers to stream the data from the Java client application to the server and/or from the server to the Java client. See <https://www.monetdb.org/downloads/Java/onclient.txt> for details. The new functionality has also been added to the JdbcClient CLI program, so you can now also use COPY ... INTO ... ON CLIENT statements from this CLI program, provided you have started it with a --csvdir argument. The CSV data files may also be gzip compressed (file name ends with: .gz). Besides this new functionality we also improved both the JDBC driver and JdbcClient to work optimal with the new MonetDB Jan2022 server release. For a list of all changes please read: <https://www.monetdb.org/downloads/Java/ChangeLog>. We recommend all users of MonetDB JDBC driver and/or JdbcClient program to upgrade to this new release. Especially if you use the new MonetDB Jan2022 server release, you should use the new jar files. The download location is as usual: <https://www.monetdb.org/downloads/Java/>. For information and tips on using the MonetDB JDBC driver, read: <https://www.monetdb.org/downloads/Java/release.txt>. Reminder: as of release monetdb-jdbc-3.0.jre8.jar the MonetDB JDBC Driver class name has been changed to: org.monetdb.jdbc.MonetDriver It used to be nl.cwi.monetdb.jdbc.MonetDriver which is marked as deprecated. Please use the new driver class name asap in your configuration files/scripts and/or Java code. The old driver class name is currently still included in the jdbc jar file but only to ease the transition for existing deployments. The old driver class will be removed in a future release of the MonetDB JDBC driver.

1 0

Announcement: New Jan2022 Feature release of MonetDB suite
by Sjoerd Mullender 24 Jan '22

24 Jan '22

The MonetDB team at MonetDB BV is pleased to announce the Jan2022 feature release of the MonetDB suite of programs. More information about MonetDB can be found on our website at <https://www.monetdb.org/>. For details on this release, please see the release notes at <https://www.monetdb.org/Downloads/ReleaseNotes>. As usual, the download location is <https://www.monetdb.org/downloads/>. ## Jan2022 Feature Release (11.43.5) ### - A couple of concurrency issues have been fixed. ### Windows Packaging - We now build Windows binaries using Visual Studio 2022. ### Client Package - A new output formatting mode was added to mclient. Use -fcsv-noquote to produce a CSV (comma-separated values) output where the quote characters have not been escapes. This can be useful when producing a single column string output that should be saved as is, e.g. when using the sys.dump_database() function. ### MonetDB Common - Implement string imprints (strimps for short) a pre-filter structure for strings in order to accelerate LIKE queries. If a strimp exists for a specific string column the strings are pre-filtered, rejecting strings that cannot possibly match, before the more expensive and accurate matching algorithms run. - On Windows, files and directories we create now get the attribute FILE_ATTIBUTE_NOT_CONTENT_INDEXED, meaning that they should not be indexed by indexing or search services. - Many (most) low level functions that could take a long time (such as BATjoin) can now be aborted with a timeout. When the function takes too long, the function will fail, and hence the whole SQL query will fail. - At some point in the past, string heaps were created where the hash value of the string was stored in the heap before the string. This hasn't been used in a long time. Now the code that could still read those old heaps has been removed. Bats that used the old format are converted automatically. - Some small interface changes to the atom functions: the atomPut function now returns (var_t) -1 on error instead of 0; the atomHeap function now returns success or failure as a gdk_return value. ### General - Reliance on the OpenSSL library has been removed. OpenSSL was used for the hash algorithms it contained (e.g. SHA512 and RIPEMD160) and for producing random numbers. The hash functions have been replaced by the original published functions, and random numbers are generated using system-specific random sources (i.e. not simply pseudo-random number generators). ### MonetDB5 Server - The storage cleanup in the 11.41.5 (Jul2021) release made the OLTP optimizer pipeline obsolete, thus it was removed. ### SQL Frontend - Add string imprints to the existing imprints index creation syntax. On string column "col" of a table "tbl" marked read only ("ALTER TABLE tbl SET READ ONLY") the user can create a string imprint using the syntax: "CREATE IMPRINTS INDEX index_name ON tbl(col);". - With the storage cleanup in the 11.41.5 (Jul2021) release, the ANALYZE statement was updated to accomodate those changes. The SAMPLE parameter is now ignored because ANALYZE generated statistics used by relational operators, are required to be precise. - In order to mitigate the I/O required to update the 'statistics' table, this table is no longer persisted. Alternately, it was changed into a computed view every time when queried. The 'stamp' and 'sample' fields were removed for the aforementioned reasons. The 'schema', 'table' and 'column' fields were added for convenience. - In previous versions there was no check that the INCREMENT BY value of a SEQUENCE was not zero. During the automatic upgrade of a database, INCREMENT BY values that are zero are set to one. - The method to compute the 'side_effect' effect property was changed for SQL functions defined in the backend engine (eg. ``CREATE FUNCTION ... EXTERNAL NAME "module"."function"''). It was changed from being computed by the SQL layer to the backend engine itself. As a consequence, the computed 'side_effect' value may be different, thus bringing incompatibilities. After an upgrade, if a 'side_effect' incompatibility arises, either the 'side_effect' value in the backend should be changed or the function should be re-created in SQL. - Removed deprecated system view sys.systemfunctions. It was marked as deprecated from release Apr2019 (11.33.3). Use query: select id as function_id from sys.functions where system; to get the same data as the old view. - Extended SQL system catalog with lookup table sys.fkey_actions and view sys.fkeys to provide user friendly querying of existing foreign keys and their ON UPDATE and ON DELETE referential action specifications. - Many improvements were done for REMOTE table plans. As a consequence, master or slave servers from this feature release are not compatible with older releases. - The view sys.ids has been changed to give more information about the objects in the system. In particular, there is an extra column added at the end that indicates whether the object is a system object. - The example modules opt_sql_append and udf are no longer loaded by default and no longer part of the binary release. If installed, they can be loaded using the --loadmodule option. - The built-in SQL functions to produce a dump that were added as a proof-of-concept in the previous release have been improved and are now usable. Use the query ``SELECT stmt FROM sys.dump_database(FALSE) ORDER BY o'' to produce a dump. The dump code built into mclient and msqldump is probably still more efficient, though. - The sys.epoch function has always been confusing. There are two versions, one with an INTEGER argument, and one with a BIGINT argument. The former accepted values as seconds, whereas the latter expected milliseconds. Also, the construct EXTRACT(EPOCH FROM value) returns a BIGINT with millisecond precision. This has now been overhauled. There is no longer a function sys.epoch with BIGINT argument, but instead there is a new function sys.epoch with DECIMAL(18,3) argument. The argument is seconds, but with 3 decimals, it provides millisecond accuracy. Also the EXTRACT(EPOCH FROM value) now returns a DECIMAL(18,3), again seconds with 3 decimals giving millisecond accuracy. Note that the internal, binary representation of DECIMAL(18,3) interpreted as seconds with 3 decimals and BIGINT with millisecond precision is exactly the same. ### Merovingian - Disabled logging into merovingian.log of next info message types: - proxying client <host>:<port> for database '<dbname>' to <url> - target connection is on local UNIX domain socket, passing on filedescriptor instead of proxying These messages were written to the log file at each connection. In most cases this information is not used. The disabling reduces the log file size. - Removed the deprecated monetdb commands `profilerstart` and `profilerstop`. ### Bug Fixes - [7168](https://github.com/MonetDB/MonetDB/issues/7168): Loosing the documentation - [7180](https://github.com/MonetDB/MonetDB/issues/7180): GROUP BY-subquery crashes MonetDb - [7182](https://github.com/MonetDB/MonetDB/issues/7182): Queries against sys.querylog_catalog, sys.querylog_calls or sys.querylog_history fail after restore of a db created using call sys.hot_snapshot(R'\path\file.tar'); - [7201](https://github.com/MonetDB/MonetDB/issues/7201): Selection of a subquery with a LEFT JOIN returns the wrong result set - [7202](https://github.com/MonetDB/MonetDB/issues/7202): DISTINCT does not work when sorting by additional columns - [7215](https://github.com/MonetDB/MonetDB/issues/7215): ODBC Driver SQLStatistics returns duplicate rows/rows for other tables

1 0

Announcement: New Jul2021-SP2 Bugfix release of MonetDB suite
by Sjoerd Mullender 21 Dec '21

21 Dec '21

The MonetDB team at MonetDB BV is pleased to announce the Jul2021-SP2 bugfix release of the MonetDB suite of programs. More information about MonetDB can be found on our website at <https://www.monetdb.org/>. For details on this release, please see the release notes at <https://www.monetdb.org/Downloads/ReleaseNotes>. As usual, the download location is <https://www.monetdb.org/downloads/>. Jul2021-SP2 Bugfix Release (11.41.13) Client Package * Dumping the database now also dumps the read-only and insert-only states of tables. MonetDB Common * Sometimes when the server was restarted, it wouldn't start anymore due to an error from BBPcheckbats. We finally found and fixed a (hopefully "the") cause of this problem. SQL Frontend * Number parsing for SQL was fixed. If a number was immediately followed by letters (i.e. without a space), the number was accepted and the alphanumeric string starting with the letter was interpreted as an alias (if aliases were allowed in that position). Bug Fixes * 7163: Multiple sql.mvc() invocations in the same query * 7167: sys.shutdown() problems * 7184: Insert into query blocks all other queries * 7185: GROUPING SETS on groups with aliases provided in the SELECT returns empty result * 7186: data files created with COPY SELECT .. INTO 'file.csv' fail to be loaded using COPY INTO .. FROM 'file.csv' when double quoted string data contains the field values delimiter character * 7191: [MonetDBe] monetdbe_cleanup_statement() with bound NULLs on variable-sized types bug * 7196: BATproject2: does not match always * 7198: Suboptimal query plan for query containing JSON access filter and two negative string comparisons * 7200: PRIMARY KEY unique constraint is violated with concurrent inserts * 7206: Python UDF fails when returning an empty table as a dictionary

1 0

New Jan2022 stable branch created
by Sjoerd Mullender 18 Nov '21

18 Nov '21

I have created a new stable branch for the next feature release. This branch was created in order to stabilize the code before the feature release. From now on *only* bug fixes may go to this new stable branch. I reserve the right to undo any changes that do not comply with this rule. I also reserve the right to admit changes that are not strictly bug fixes. The name for the branch is Jan2022. So in each of your stable checkout directories you can do hg pull hg update -rJan2022 to update to this new branch. -- Sjoerd Mullender

1 0

Announcement: New Jul2021-SP1 Bugfix release of MonetDB suite
by Sjoerd Mullender 05 Oct '21

05 Oct '21

The MonetDB team at MonetDB BV is pleased to announce the Jul2021-SP1 bugfix release of the MonetDB suite of programs. More information about MonetDB can be found on our website at <https://www.monetdb.org/>. For details on this release, please see the release notes at <https://www.monetdb.org/Downloads/ReleaseNotes>. As usual, the download location is <https://www.monetdb.org/downloads/>. Jul2021-SP1 Bugfix Release (11.41.11) MonetDB Common * Some deadlock and race condition issues were fixed. * Handling of the list of free bats has been improved, leading to less thread contention. * A problem was fixed where the server wouldn't start with a message from BBPcheckbats about files being too small. The issue was not that the file was too small, but that BBPcheckbats was looking at the wrong file. * An issue was fixed where a "short read" error was produced when memory was getting tight. * When appending to a string bat, we made an optimization where the string heap was sometimes copied completely to avoid having to insert strings individually. This copying was still done too eagerly, so now the string heap is copied less frequently. In particular, when appending to an empty bat, the string heap is now not always copied whole. SQL Frontend * If the server has been idle for a while with no active clients, the write-ahead log is now rotated. * A problem was fixed where files belonging to bats that had been deleted internally were not cleaned up, leading to a growing database (dbfarm) directory. * A leak was fixed where extra bats were created but never cleaned up, each taking up several kilobytes of memory. * [This feature was already released in Jul2021 (11.41.5), but the ChangeLog was missing] Grant indirect privileges. With "GRANT SELECT ON <my_view> TO <another_user>" and "GRANT EXECUTE ON FUNCTION <my_func> TO <another_user>", one can grant access to "my_view" and "my_func" to another user who does not have access to the underlying database objects (e.g. tables, views) used in "my_view" and "my_func". The grantee will only be able to access data revealed by "my_view" or conduct operations provided by "my_func". * Improved error reporting in COPY INTO by giving the line number (starting with one) for the row in which an error was found. In particular, the sys.rejects() table now lists the line number of the CSV file on which the record started in which an error was found. Bug Fixes * 7140: SQL Query Plan Non Optimal with View * 7162: Extend sys.var_values table * 7165: `JOINIDX: missing '.'` when running distributed join query on merged remote tables * 7172: Unexpected query result with merge tables * 7173: If truncate is in transaction then after restart of MonetDB the table is empty * 7178: Remote Table Throws Error - createExceptionInternal: !ERROR: SQLException:RAstatement2:42000!The number of projections don't match between the generated plan and the expected one: 1 != 1200

1 0

Announcement: New Jul2021 Feature release of MonetDB suite
by Sjoerd Mullender 09 Aug '21

09 Aug '21

The MonetDB team at MonetDB BV is pleased to announce the Jul2021 feature release of the MonetDB suite of programs. More information about MonetDB can be found on our website at <https://www.monetdb.org/>. For details on this release, please see the release notes at <https://www.monetdb.org/Downloads/ReleaseNotes>. As usual, the download location is <https://www.monetdb.org/downloads/>. Jul2021 Feature Release (11.41.5) Client Package * The MonetDB stethoscope has been removed. There is now a separate package available with PIP (monetdb_stethoscope) or as an RPM or DEB package (stethoscope) from the monetdb.org repository. Mapi Library * Add optional MAPI header field which can be used to immediately set reply size, autocommit, time zone and some other options, see mapi.h. This makes client connection setup faster. Support has been added to mapilib, pymonetdb and the jdbc driver. ODBC Driver * A typo that made the SQLSpecialColumns function unusable was fixed. MonetDB Common * A bug in the grouping code has been fixed. * Hash indexes are no longer maintained at all cost: if the number of distinct values is too small compared to the total number of values, the index is dropped instead of being maintained during updates. * A new type, called msk, was introduced. This is a bit mask type. In a bat with type msk, each row occupies a single bit, so 8 rows are stored in a single byte. There is no NULL value for this type. * The function of the BAT iterator (type BATiter, function bat_iterator) has been expanded. The iterator now contains more information about the BAT, and it contains a pointer to the heaps (theap and tvheap) that are stable, at least in the sense that they will remain available even when parallel threads update the BAT and cause those heaps to grow (and therefore possibly move in memory). A call to bat_iterator must now be accompanied by a call to bat_iterator_end. * Implemented function BUNreplacemultiincr to replace multiple values in a BAT in one go, starting at a given position. * Implemented new function BUNreplacemulti to replace multiple values in a BAT in one go, at the given positions. * Removed function BUNinplace, just use BUNreplace, and check whether the BAT argument is of type TYPE_void before calling if you don't want to materialize. * Implemented a function BUNappendmulti which appends an array of values to a BAT. It is a generalization of the function BUNappend. * Changed the interface of the atom read function. It now requires an extra pointer to a size_t value that gives the current size of the destination buffer, and when that buffer is too small, it receives the size of the reallocated buffer that is large enough. In any case, and as before, the return value is a pointer to the destination buffer. * Environment variables (sys.env()) must be UTF-8, but since they can contain file names which may not be UTF-8, there is now a mechanism to store the original values outside of sys.env() and store %-escaped (similar to URL escaping) values in the environment. The key must still be UTF-8. * We now save the location of the min and max values when known. MonetDB5 Server * When using the --in-memory option, mserver5 will run completely in memory, i.e. not create a database on disk. The server can still be connected to using the name of the in-memory database. This name is "in-memory". * By using the option "--dbextra=in-memory", mserver5 can be instructed to keep transient BATs completely in memory. SQL Frontend * The system view sys.ids has been updated to include some more system IDs. * The sys.storage() function now only returns meta data, i.e. data that can be calculated without access to the column contents. * Since STREAM tables support is removed, left over STREAM tables are dropped from the catalog. * Fix a warning emitted by some implementations of the tar(1) command when unpacking hot snapshot files. * support reading the concatenation of compressed files as a single compressed file. * COPY BINARY overhaul. Allow control over binary endianness using COPY [ (BIG | LITTLE | NATIVE) ENDIAN] BINARY syntax. Defaults to NATIVE. Strings are now \0 terminated rather than \n. Support for BOOL, TINYINT, SMALLINT, INT, LARGEINT, HUGEINT, with their respective "INTMIN" values as the NULL representation; 32 and 64 bit FLOAT/REAL, with NaN as the NULL representation; VARCHAR/TEXT, JSON and URL with \x80 as the NULL representation; UUID as fixed width 16 byte binary values, with (by default) all zeroes as the NULL representation; temporal type structs as defined in copybinary.h with any invalid value as the NULL representation. * In the Jul2021 release the storage and transaction layers have undergone major changes. The goal of these changes is robust performance under inserts/updates and deletes and lowering the transaction startup costs, allowing faster (small) queries. Where the old transaction layer duplicated a lot of data structures during startup, the new layer shares the same tree. Using object timestamps the isolation of object is guaranteed. On the storage side the timestamps indicate whether a row is visible (deleted or valid), to a transaction as well. The changes also give some slight changes on the perceived transactional behavior. The new implementation uses shared structures among all transactions, which do not allow multiple changes of the same object. And we then follow the principle of the first writer wins, i.e., if a transaction creates a table with name 'table_name', and concurrently one other transaction does the same the later of the two will fail with a concurrency conflict error message (even if the first writer never commits). We expect most users not to notice this change, as such schema changes aren't usually done concurrently. * There is now a function sys.current_sessionid() to return the session ID of the current session. This ID corresponds with the sessionid in the sys.queue() result. * Merge statements could not produce correct results on complex join conditions, so a renovation was made. As a consequence, subqueries now have to be disabled on merge join conditions. * preserve in-query comments * Use of CTEs inside UPDATE and DELETE statements are now more restrict. Previously they could be used without any extra specification in the query (eg. with "v1"("c1") as (...) delete from "t" where "t"."c1" = "v1"."c1"), however this was not conformant with the SQL standard. In order to use them, they must be specified in the FROM clause in UPDATE statements or inside a subquery. * Added 'schema path' property to user, specifying a list of schemas to be searched on to find SQL objects such as tables and functions. The scoping rules have been updated to support this feature and it now finds SQL objects in the following order: 1. On occasions with multiple tables (e.g. add foreign key constraint, add table to a merge table), the child will be searched on the parent's schema. 2. For tables only, declared tables on the stack. 3. 'tmp' schema if not listed on the 'schema path'. 4. Session's current schema. 5. Each schema from the 'schema path' in order. 6. 'sys' schema if not listed on the 'schema path'. Whenever the full path is specified, ie "schema"."object", no search will be made besides on the explicit schema. * To update the schema path ALTER USER x SCHEMA PATH y; statement was added. [SCHEMA PATH string] syntax was added to the CREATE USER statement. The schema path must be a single string where each schema must be between double quotes and separated with a single comma, e.g. '"sch1","sch2"' For every created user, if the schema path is not given, '"sys"' will be the default schema path. * Changes in the schema path won't be reflected on currently connected users, therefore they have to re-connect to see the change. Non existent schemas on the path will be ignored. * Leftover STREAM table definition from Datacell extension was removed from the parser. They had no effect anymore. Merovingian * Deprecate `profilerstart` and `profilerstop` commands. Since stethoscope is a separate project (https://github.com/MonetDBSolutions/monetdb-pystethoscope) the installation directory is not standard anymore. `profilerstart` and `profilerstop` commands assume that the stethoscope executable is in the same directory as `mserver5`. This is no longer necessarily true since stethoscope can now be installed in a python virtual environment. The commands still work if stethoscope is installed using the official MonetDB installers, or if a symbolic link is created in the directory where `mserver5` is located. * The exittimeout value can now be set to a negative value (e.g. -1) to indicate that when stopping the dbfarm (using monetdbd stop dbfarm), any mserver5 processes are to be sent a termination signal and then waited for until they terminate. In addition, if exittimeout is greater than zero, the mserver5 processes are sent a SIGKILL signal after the specified timeout and the managing monetdbd is sent a SIGKILL signal after another five seconds (if it didn't terminate already). The old situation was that the managing monetdbd process was sent a SIGKILL after 30 seconds, and the mserver5 processes that hadn't terminated yet would be allowed to continue their termination sequence. Bug Fixes * 2030: Temporary table is semi-persistent when transaction fails * 7031: I cannot start MoentDb, because the installation path has Chinese. * 7055: Table count returning function used inside other function gives wrong results. * 7075: Inconsistent Results using CTEs in Large Queries * 7079: WITH table AS... UPDATE ignores the WHERE conditions on table * 7081: Attempt to allocate too much space in UPDATE query * 7093: `current_schema` not in sys.keywords * 7096: DEBUG SQL statement broken * 7115: Jul2021: ParseException while upgrading Oct2020 database * 7116: Jul2021: Cannot create filter functions * 7125: MonetDB Round Function issues in the latest release * 7126: The "lower" and "upper" functions doesn't work for Cyrillic alphabet * 7127: Bug report: "write error on stream" that results in mclient crash * 7128: Bug report: strange error message "Subquery result missing" * 7129: Bug report: TypeException:user.main[19]:'batcalc.between' undefined * 7130: Bug report: TypeException:user.main[396]:'algebra.join' undefined * 7131: Bug report: TypeException:user.main[273]:'bat.append' undefined * 7133: WITH <alias> ( SELECT x ) DELETE FROM ... deletes wrong tuples * 7136: MERGE statement is deleting rows if the column is set as NOT NULL even though it should not * 7137: Segmentation fault while loading data * 7138: Monetdb Python UDF crashes because of null aggr_group_arr * 7141: COUNT(DISTINCT col) does not calculate correctly distinct values * 7142: Aggregates returning tables should not be allowed * 7144: Type up-casting (INT to BIGINT) doesn't always happen automatically * 7146: Query produces this error: !ERROR: Could not find %102.%102 * 7147: Internal error occurs and is not shown on the screen * 7148: Select distinct is not working correctly * 7151: Insertion is too slow * 7153: System UDFs lose their indentation - Python functions broken * 7158: Python aggregate UDF returns garbage when run on empty table * 7161: fix priority

1 0

New MonetDB JDBC driver released
by Sjoerd Mullender 16 Jun '21

16 Jun '21

The MonetDB team at MonetDB BV is pleased to announce a bugfix release of the MonetDB Java jars and JDBC driver. We recommend all users of MonetDB JDBC driver to upgrade to version: monetdb-jdbc-3.1.jre8.jar The download location is as usual: <https://www.monetdb.org/downloads/Java/>. For a list of changes please read: <https://www.monetdb.org/downloads/Java/ChangeLog>. For information and tips on using the MonetDB JDBC driver, see: <https://www.monetdb.org/downloads/Java/release.txt>. Note: as of release monetdb-jdbc-3.0.jre8.jar the MonetDB JDBC Driver class name has been changed to: org.monetdb.jdbc.MonetDriver It used to be nl.cwi.monetdb.jdbc.MonetDriver which is now marked as deprecated. Please use the new driver class name asap in your configuration files/scripts and/or Java code. The old driver class name is still included in the jar file to ease the transition for existing deployments. However it will be removed in a future release of the MonetDB JDBC driver. -- Sjoerd Mullender

1 0

sys.queue.query fields do not contain original query sent to the server
by Martin van Dinther 20 May '21

20 May '21

Hi developers, When viewing the queries sent to the server via SQL: select * from queue order by tag desc; I noticed that the column "query" does NOT lists the original queries sent to the server, but changed ones. It has replaced all single quote characters ' by prefixing them with a backslash \, so \' Also it appends a \n; after the SQL statement when no ; character was present at the end. Both changes are undesirable and unneeded. Due to the added \ characters the content of the query field can no longer be copied into an SQL client (such as mclient or JdbcClient or SQuirreL) and rerun easily. All the \ characters must be manually removed, which is very annoying and error-prone. Why isn't the original query received by the server stored and returned by sys.queue.query? Can it be changed to only store the exact received SQL query string, so without any changes/alterations made to the query string? Also no adding of \n; characters. I am using latest release MonetDB Oct2020-SP5. Regards, Martin

1 0

Announcement: New Oct2020-SP5 Bugfix release of MonetDB suite
by Sjoerd Mullender 10 May '21

10 May '21

The MonetDB team at MonetDB BV is pleased to announce the Oct2020-SP5 bugfix release of the MonetDB suite of programs. More information about MonetDB can be found on our website at <https://www.monetdb.org/>. For details on this release, please see the release notes at <https://www.monetdb.org/Downloads/ReleaseNotes>. As usual, the download location is <https://www.monetdb.org/downloads/>. Oct2020-SP5 Bugfix Release (11.39.17) Windows Packaging * Added the monetdbe library to the Windows installer. MonetDB Common * A bug that would very occasionally produce an error "strPut: incorrectly encoded UTF-8", even when no incorrectly coded characters are used at all, has been fixed. It was the result of a rare combination of strings having been added to the column that caused essentially an off-by-one type of error to occur. SQL Frontend * A bug where a sequence of TRUNCATE TABLE and COPY INTO the just truncated table would result in success being reported to both queries, but the table still being empty afterwards, has been fixed. Merovingian * When stopping monetdbd using the `monetdbd stop' command, this command now waits for 5 seconds longer than the exittimeout value before it kills the monetdbd daemon instead of only 30 seconds total (or until that daemon stops earlier). This gives the daemon enough time to terminate the mserver5 processes that it is managing. If exittimeout is negative, the daemon and the monetdbd process initiating the stop wait indefinitely until the mserver5 processes have stopped. Bug Fixes * 3336: DB files not removed if all rows are deleted, even after restart * 7104: Monetdbe NTILE function does not produce correct ordering * 7108: Monetdb crashes on query execution * 7109: MERGE Statement incorrectly reports that input relation matches multiple rows * 7110: Monetdb Query parsing consistency issues in the latest release (Remote Table)

1 0

Announcement: New Oct2020-SP4 Bugfix release of MonetDB suite
by Sjoerd Mullender 12 Apr '21

12 Apr '21

The MonetDB team at MonetDB BV is pleased to announce the Oct2020-SP4 bugfix release of the MonetDB suite of programs. More information about MonetDB can be found on our website at <https://www.monetdb.org/>. For details on this release, please see the release notes at <https://www.monetdb.org/Downloads/ReleaseNotes>. As usual, the download location is <https://www.monetdb.org/downloads/>. Oct2020-SP4 Bugfix Release (11.39.15) ODBC Driver * When connecting using a DSN (Data Source Name), information about the data source is retrieved from the ODBC.INI file. Now we also get the location of the LOGFILE from this file. The logfile can be used to log all calls to the MonetDB ODBC driver to a file which can be used for debugging. * The ODBC driver now only passes on information about HUGEINT columns as HUGEINT when the application has indicated interest by querying about the SQL_HUGEINT extension type using the SQLGetTypeInfo function or by specifying the type in a call to SQLSetDescField. Otherwise the driver silently translates the HUGEINT type to BIGINT. This means that most application will see BIGINT columns when the server produced a HUGEINT column and only give an error if the value in the HUGEINT column didn't fit into a BIGINT. Bug Fixes * 6786: function json.isvalid(js json) is not useful, could be removed * 7016: Database crashes when use similarity function on a table with more than 200k records * 7037: Clearer err msg for ALTER USER with insufficient privileges * 7042: AddressSanitizer:DEADLYSIGNAL in Oct2020/gdk/gdk_tracer.c:494 * 7050: file descriptor leak when forward=redirect * 7057: ODBC driver installer on Windows is missing some DLLs * 7058: MonetDBe: COPY INTO csv file does not produce any output * 7059: MonetDBe: 'reverse' C UDF crashes * 7061: Have bulk load support combined gzip files * 7064: Temporary hashes created in hash and unique logic should try to use transient data farm first * 7066: percent_rank function with wrong results * 7070: double free error when running MonetDBe Example * 7076: mserver5 ignores memory.low from cgroups v2 * 7077: Oct2020: new default privileges not effectively communicated * 7083: MonetDBe C++ Compiling Error * 7085: Mitosis and filter functions * 7087: SIGSEGV caused by error in subquery's function being ignored by top-level query * 7089: Data consistency problem of query results in the latest release of Monetdb (Remote Table)

1 0

monetdbe, fetching date, daytime, timestamp in their MonetDB native memory storage format
by Daniel Zvinca 02 Mar '21

02 Mar '21

Hello, Since the first version of monetdblite (monetdbe now), passing/fetching the date and time columns (in monetdbe or capi code for instance) is done using intermediary structures, instead of native types (int, __int64). typedef struct { unsigned char day; unsigned char month; short year; } monetdbe_data_date; typedef struct { unsigned int ms; unsigned char seconds; unsigned char minutes; unsigned char hours; } monetdbe_data_time; typedef struct { monetdbe_data_date date; monetdbe_data_time time; } monetdbe_data_timestamp; typedef struct { unsigned char day; unsigned char month; int year; } cudf_data_date; typedef struct { unsigned int ms; unsigned char seconds; unsigned char minutes; unsigned char hours; } cudf_data_time; typedef struct { cudf_data_date date; cudf_data_time time; } cudf_data_timestamp; I am wondering if this conversion is truly necessary, considering that this effort is usually doubled by either converting back to native format (for append operations, for instance) or extra conversion in other flavours of date/time structures some other C code might need. The very few macros required for time information extraction from the native types (is_nil, extract) can be done on the client side when needed. Related to this, a low level function "monetdbe_result_scan" alternative to "monetdbe_result_fetch" would avoid extra memory allocation in case no storage is required or if a different storage for the result (user list, compressed, straight on disk, ...) is considered by a developer. For integers, doubles, float, date, daytime, timestamp types passing the data address Tloc(b, 0) in the same way GENERATE_BAT_INPUT is doing it, looks good enough, assuming the developer knows to use that memory address for reading purposes. For the rest of the types (str, blob, others), invoking a pointer to a provided user function (void *data, size_t data_size, size_t j) inside the BATloop would suffice. This "monetdbe_result_scan" part can be adjusted by a custom development, sure, but passing date-time types columns in native format (which can come from a remote connection as well) should be uniform. Having a version ID for the monetdbe_result structure would also help in case a remote connection can be instructed to provide results in different ways. Dan

2 1

Announcement: New Oct2020-SP3 Bugfix release of MonetDB suite
by Sjoerd Mullender 19 Feb '21

19 Feb '21

The MonetDB team at MonetDB BV is pleased to announce the Oct2020-SP3 bugfix release of the MonetDB suite of programs. More information about MonetDB can be found on our website at <https://www.monetdb.org/>. For details on this release, please see the release notes at <https://www.monetdb.org/Downloads/ReleaseNotes>. As usual, the download location is <https://www.monetdb.org/downloads/>. Oct2020-SP3 Bugfix Release (11.39.13) Bug Fixes * 7049: Implement DISTINCT for GROUP_CONCAT

1 0

Announcement: New Oct2020-SP2 Bugfix release of MonetDB suite
by Sjoerd Mullender 25 Jan '21

25 Jan '21

The MonetDB team at MonetDB BV is pleased to announce the Oct2020-SP2 bugfix release of the MonetDB suite of programs. More information about MonetDB can be found on our website at <https://www.monetdb.org/>. For details on this release, please see the release notes at <https://www.monetdb.org/Downloads/ReleaseNotes>. As usual, the download location is <https://www.monetdb.org/downloads/>. Oct2020-SP2 Bugfix Release (11.39.11) SQL Frontend * CREATE [OR REPLACE] TRIGGER schema_name.trigger_name is now disallowed, because the trigger will be stored on the same schema as the table it refers to. Use a schema-qualified on the table reference (ie ON clause) when necessary. Bug Fixes * 6862: mserver5: crashes under update_table() when calling lib_sql.so ( max_clients = 2048) * 7002: monetdb stop fails * 7012: mclient enters an infinite loop when a file on the command line does not exist * 7013: Select * on grouped view: wrong error "cannot use non GROUP BY column 'a1' in query results without an aggregate function" * 7017: mal seems to leak in functions * 7020: release an older savepoint causes "BATproject2: does not match always" * 7021: savepoints crash mserver5 * 7022: transaction with an unreleased savepoint not properly persisted * 7023: CREATE VIEW: SELECT: cannot use non GROUP BY column '%1' in query results without an aggregate function * 7024: DELETE FROM or TRUNCATE on freshly created table leads to loosing all further inserts in same transaction * 7030: DROP TABLE with AUTO_INCREMENT doesn't drop sequence causing left-over dependency * 7034: User with sysadmin role cannot create another user * 7035: UPDATE and SELECT column privileges * 3772: Any user can grant a role.

1 0

MonetDB bug tracker has moved to github
by Sjoerd Mullender 01 Dec '20

01 Dec '20

The MonetDB bug tracker has been moved from our self-hosted bugzilla instance to github issues. All bug IDs have remained the same in the transition, making referencing bugs easier. The new bug tracker can be found at https://github.com/MonetDB/MonetDB/issues Don't forget to follow this tracker (if you're interested in doing that) or follow individual bugs. Due to the way we had to do the move, the old stars that you may have given to the MonetDB github repo have been lost, so please, star it again (or for the first time) if you feel so inclined. The old bugzilla tracker will remain on-line for the time being but entry of new bugs has been turned off. Shortcuts to bugs are forwarded to github. (Shortcuts are URLs in the form of https://www.monetdb.org/bugzilla/<bugid> and https://bugs.monetdb.org/<bugid>.) -- Sjoerd Mullender

1 0

MonetDB bug tracker is going to move
by Sjoerd Mullender 29 Nov '20

29 Nov '20

LS, The MonetDB team wishes to announce the end of our reliance on Bugzilla for bug tracking and enhancement requests. Instead, we switch over to the MonetDB Github issues repo (https://github.com/monetdb/monetdb) which is an hourly based mirror of the Mercurial repository managed at the development group. The move will take place on Friday, November 27, 2020. The actual transfer of the Bugzilla archive into the Github Issues may take a number of hours. A redirect will be in place to send people from Bugzilla towards the MonetDB GitHub repository instead. -- Sjoerd Mullender

1 1

Announcement: New Oct2020-SP1 Bugfix release of MonetDB suite
by Sjoerd Mullender 24 Nov '20

24 Nov '20

The MonetDB team at MonetDB BV is pleased to announce the Oct2020-SP1 bugfix release of the MonetDB suite of programs. More information about MonetDB can be found on our website at <https://www.monetdb.org/>. For details on this release, please see the release notes at <https://www.monetdb.org/Downloads/ReleaseNotes>. As usual, the download location is <https://www.monetdb.org/downloads/>. Oct 2020-SP1 bugfix release (11.39.7) Bug Fixes * 6890: Add support of xz/lzma (de)compression on MS Windows * 6891: Add support of lz4 (de)compression on MS Windows * 6971: Parsing table returning function on remote server fails * 6981: Oct2020: PREPARE DDL statement silently fails * 6983: monetdb allows to use non-existing optimizer pipe * 6998: MAL profiler buffer limitations * 7001: crossproduct generated for a simple (semi-)join * 7003: Segfault on large chain of constant decimal multiplication * 7005: Dropping a STREAM TABLE does not remove the associated column info from sys._columns * 7010: deallocate <id> results in all prepared statements being deallocated (not error-related) * 7011: uuid() called only once when used in projection list

1 0

Announcement: New Oct2020 Feature release of MonetDB suite
by Sjoerd Mullender 19 Oct '20

19 Oct '20

The MonetDB team at MonetDB BV is pleased to announce the Oct2020 feature release of the MonetDB suite of programs. More information about MonetDB can be found on our website at <https://www.monetdb.org/>. For details on this release, please see the release notes at <https://www.monetdb.org/Downloads/ReleaseNotes>. As usual, the download location is <https://www.monetdb.org/downloads/>. Oct 2020 feature release (11.39.5) MonetDB5 Server * The settings for specifying how mserver5 should listen to "The Internet" have been overhauled. See the manual for details. In brief, mapi_autosense, mapi_ipv6 and mapi_open are gone. If mapi_listenaddr equals "localhost" or "all", we listen to both IPv4 and IPv6 (if available), if "127.0.0.1" or "0.0.0.0", we listen to IPv4 only, if "::1" or "::" we listen to IPv6 only. The first of each pair is loopback interface only, the second is all interfaces. If mapi_listenaddr is "none", then no IP port is opened, you need to use a UNIX domain socket. If mapi_port is 0, we let the operating system choose a free port (like mapi_autosense). Default behavior has not changed. Whole System * Finished a first version of the new monitoring function user_statistics(), which is only intended for the DBAs. For each database user who has logged in during the current mserver5 session, it returns "username": login name of the database user, "querycount": the number of queries this user has executed since his/her first login, "totalticks": the total execution time (in microsecond) of the queries ran by this user, "maxquery": the query with the longest execution time (if two queries have the same execution time, the newer overwrites the older), "maxticks": the execution time of the 'maxquery' (in microsecond), "started": the start timestamp of the 'maxquery', "finished": the finish timestamp of the 'maxquery'. * Removed support for LiDAR data, that is the SQL procedures sys.lidarattach, sys.lidarload, and sys.lidarexport. * The sys.queue() has been turned into a circular buffer to allow for inspection of both active, paused and recently executed queries. Client Package * mclient and msqldump now also look in $XDG_CONFIG_HOME for the monetdb configuration file. SQL Frontend * Extended the system monitor with a table-returning function user_statistics() which keeps some statistics for each SQL user, e.g. the user's query count, total time spent, and maximal query seen. * Removed '@' syntax used to refer into a variable in a query. It was a non-standard method, which was replaced by a schema addition to variables. Variables in the global scope now have schema. All default global variables are set under schema "sys". However variables inside PSM don't have a schema, because there are no transaction semantics inside PSM at the moment. * Removed declared variables and tables from the global scope. They were transaction agnostic and incompatible with the SQL standard, i.e. they are valid exclusively under PSM (e.g. functions, procedures and triggers). * Scoping semantics were added for both variables and tables. Variables with the same name at a query are now resolved under the following precedence rules: 1. Tables, Views and CTEs at the FROM clause. 2. Variable declared in the body of function/procedure, i.e. local variable. 3. Function/procedure parameter. 4. Variable from the global scope. Tables with the same name now have the following precedence rules at a SQL query: 1. Table declared in the body of function/procedure, ie local table. 2. Temporary table. 3. Table from the current session schema. This means the query: SELECT * FROM "keys"; will list keys from temporary tables instead of persisted ones, because "keys" table is available for both "sys" and "tmp" schemas. * The table returning function "var" was extended with more details about globally declared variables, namely their schema, type and current value. * Removed obsolete sys.getContent(url) function. * Removed obsolete json.output(json) function. * Removed compatibility between interval types and other numeric types in favor for a more strict SQL standard compliance. This means operations between temporal types and other numeric types such as INT and DECIMAL are no longer possible, instead use interval types. e.g. SELECT date '2020-01-01' + 1; now gives the error. Instead do: SELECT date '2020-01-01' + interval '1' day; if 1 was meant to be a day interval. Setting an interval variable such as the session's current timezone with a number e.g. SET current_timezone = 1; is no longer possible. Instead do SET current_timezone = interval '1' hour; Casting between interval and other numeric types is no longer possible as well, because they are not compatible. * Because of incompatibilities this change may create, if a user intends to convert a numeric value to an interval, the multiplication function can be used in the form: <numeric value> * interval '1' <interval length> e.g. 10 * interval '1' second = interval '10' second. As for the other way around, the 'EPOCH' option was added to the extract syntax. This option returns the number of milliseconds since the UNIX epoch 1970-01-01 00:00:00 UTC for date, timestamp and time values (it can be negative). Meanwhile, for day and second intervals, it returns the total number of milliseconds in the interval. As a side note, the 'EPOCH' option is not available for month intervals, because this conversion is not transparent for this type. * Made general logarithm function log(x,base) compliant with the SQL standard, by swapping the input parameters. Instead of log(x,base), now is log(base,x). Bug Fixes * 3553: All schema access to ubiquitous functions * 3815: Incorrect results when expression contains implicit float/integer conversions * 6415: Date arithmetic types are inconsistent * 6814: provide native implementations for scalar functions sys.degrees(rad) and sys.radians(deg) * 6843: function sys.getcontent(url) always returns "Feature not supported" * 6857: remove not implemented aggregate function json.output(js json) * 6870: Missing bulk operators * 6878: SQL Connection Error when running SELECT queries containing AND command * 6910: SQLancer query: 'bat.append' undefined * 6930: SQLancer crash on join with coalesce * 6931: Allow EDITOR to be used for the current command in mclient * 6935: Wrong result when dividing interval by literal float * 6937: Lost the microsecond precisions * 6938: Segmentation fault in MalOptimizer * 6939: Error in optimizer multiplex when selecting profiler.getlimit() or wlc.clock() or wlc.tick() or wlr.clock() or wlr.tick() * 6941: SELECT queries on remote table fail when using LIKE in WHERE conditions * 6943: JSON parser is too permissive * 6948: msqldump with Empty BLOBs cannot be imported * 6949: Loosing timing precision * 6950: redundant/replicated code line in gdk/gdk_hash.c * 6951: Use a different naming scheme for MAL blocks * 6954: FILTER functions no longer find their implementation * 6955: ROUND(DECIMAL, PRECISION) gives incorrect result with non-scalar precision parameter * 6960: implementation of log(arg1,arg2) function is not compliant with the SQL standard, arguments are switched * 6962: "SELECT * FROM ids LIMIT 1" produces: exp_bin: !ERROR: Could not find %173.id * 6964: Table returning function: Cannot access column descriptor * 6965: Crash when using distinct on the result of a table returning function * 6974: Oct2020-branch cannot attach and load FITS files * 6976: Oct2020: default dbfarm cannot be started * 6978: Oct2020: \d shows empty result in schema created by include sql script * 6979: timestamp add integer * 6980: Oct2020: wrong mel definition for str.epilogue

1 0

New Oct2020 stable branch created
by Sjoerd Mullender 21 Aug '20

21 Aug '20

I have created a new stable branch for the next feature release. This branch was created in order to stabilize the code before the feature release. >From now on *only* bug fixes may go to this new stable branch. I reserve the right to undo any changes that do not comply with this rule. I also reserve the right to admit changes that are not strictly bug fixes. The name for the branch is Oct2020. So in each of your stable checkout directories you can do hg pull hg update -rOct2020 to update to this new branch. -- Sjoerd Mullender

1 0

Possible leaks (MonetDB-11.37.7\common\stream\stream.c)
by Ranier Vilela 04 Aug '20

04 Aug '20

Possible leaks: MonetDB-11.37.7\common\stream\stream. 1. At line 1306 if (stream_bzread(s, buf, 1, UTF8BOMLENGTH) == UTF8BOMLENGTH && strncmp(buf, UTF8BOM, UTF8BOMLENGTH) == 0) { s->isutf8 = true; } else if (s->stream_data.p) { bzp = s->stream_data.p; BZ2_bzReadClose(&err, bzp->b); rewind(bzp->f); bzp->b = BZ2_bzReadOpen(&err, bzp->f, 0, 0, NULL, 0); } If stream_bzread at line 1302 is successful, and (s->stream_data.p) is not null at line 1305, bzp (original pointer malloc) is not lost at line 1306? bzp = s->stream_data.p; 2. At line 1322: if (err != BZ_OK) { stream_bzclose(s); destroy(s); return NULL; } miss free(bzp);? destroy(s) do not free bzp, according, with line 1283. 3. At line 2350 miss free(c); regards, Ranier Vilela

2 4

Possible type with bitwise operator (&)
by Ranier Vilela 03 Aug '20

03 Aug '20

Hi, At gdk\gdk_bat.c (line 1384) variable b->type is bool Would be a typo? if (((b->ttype != TYPE_void) & b->tkey) && b->batCount > 1) { Should be: if (((b->ttype != TYPE_void) && b->tkey) && b->batCount > 1) { regards, Ranier Vilela

4 16

Possible typo with bitwise operator (&)
by Ranier Vilela 01 Aug '20

01 Aug '20

Hi, At gdk\gdk_bat.c (line 1384) variable b->type is bool Would be a typo? if (((b->ttype != TYPE_void) & b->tkey) && b->batCount > 1) { Should be: if (((b->ttype != TYPE_void) && b->tkey) && b->batCount > 1) { regards, Ranier Vilela

1 0

Announcement: New Jun2020-SP1 Bugfix release of MonetDB suite
by Sjoerd Mullender 27 Jul '20

27 Jul '20

The MonetDB team at MonetDB BV is pleased to announce the Jun2020-SP1 bugfix release of the MonetDB suite of programs. More information about MonetDB can be found on our website at <https://www.monetdb.org/>. For details on this release, please see the release notes at <https://www.monetdb.org/Downloads/ReleaseNotes>. As usual, the download location is <https://www.monetdb.org/downloads/>. Jun 2020-SP1 bugfix release (11.37.11) MonetDB Common * Hash buckets come in variable widths. But if a BAT grows long enough so that the BAT indexes that are stored in the buckets don't fit anymore, the buckets need to be widened. This is now fixed. Bug Fixes * 6844: sys.getUser('https://me:pw@www.monetdb.org/Doc') does not return the user: me * 6845: the url sys.get...(url) functions do not allow null as a parameter * 6858: json.keyarray(json '{ "":0 }') fails with error: Could not allocate space * 6859: only first character of the separator string in json.text(js json, sep string) is used * 6873: sys.hot_snapshot() creates incomplete snapshots if the write-ahead log is very large * 6876: tar files created by sys.hot_snapshot() produce warnings on some implementations of tar * 6877: MonetDB produces malformed LZ4 files * 6880: Left fuzzy queries are much slower than other fuzzy queries. * 6882: cgroups limits no longer respected? * 6883: SQLancer crash on delete query * 6884: SQLancer generates query with unclear error message * 6885: SQLancer causes assertion error on UTF8_strlen * 6886: SQLancer alter table add unique gives strange error message * 6887: SQLancer crash on complex query * 6888: SQLancer crash on cross join on view * 6889: SQLancer crash on long query * 6892: SQLancer crash on query with HAVING * 6893: SQLancer inner join reporting GDK error * 6894: SQLancer crash on rtrim function * 6895: SQLancer causing 'algebra.select' undefined error * 6896: SQLancer algebra.select' undefined 2 * 6897: SQLancer distinct aggregate with error on group by constant * 6898: SQLancer crash on join query * 6899: SQLancer TLP query with wrong results * 6900: SQLancer generated SIGFPE * 6901: SQLancer TLP query with wrong results 2 * 6902: SQLancer query: batcalc.between undefined * 6903: SQLancer calc.abs undefined * 6904: SQLancer aggr.subavg undefined * 6905: SQLancer TLP query with wrong results 3 * 6906: SQLancer crash on complex join * 6907: SQLancer algebra.select undefined * 6908: SQLancer inputs not the same size * 6909: SQLancer query with wrong results * 6911: SQLancer query: 'calc.bit' undefined * 6917: Decimal parsing fails * 6918: SQLancer query compilation error * 6919: SQLancer insert function doesn't handle utf-8 strings * 6920: SQLancer project_bte: does not match always * 6922: Timestamp columns not migrated to new format * 6923: Imprints data files for timestamp BAT not migrated to the new format * 6924: SQLancer query copy on unique pair of columns fails and complex query with GDK error * 6925: Count string rows in union of string tables leaks (RSS) memory * 6926: SQLancer query with wrong results * 6927: SQLancer inputs not the same size * 6928: SQLancer crash on coalesce * 6929: SQLancer calc.date undefined * 6932: Syntax error while parsing JSON numbers with exponent * 6934: sys.isauuid() returns wrong answer for some invalid uuid strings

1 0

Announcement: New Jun2020 Feature release of MonetDB suite
by Sjoerd Mullender 03 Jun '20

03 Jun '20

The MonetDB team at MonetDB BV is pleased to announce the Jun2020 feature release of the MonetDB suite of programs. More information about MonetDB can be found on our website at <https://www.monetdb.org/>. For details on this release, please see the release notes at <https://www.monetdb.org/Downloads/ReleaseNotes>. As usual, the download location is <https://www.monetdb.org/downloads/>. Jun 2020 feature release (11.37.7) MonetDB5 Server * The mserver5 option --verbose (-v) was removed. A similar effect can be had by issuing the query CALL logging.setcomplevel('SQL_TRANS', 'INFO'); as the monetdb user. * The functions algebra.intersect, algebra.semijoin, and algebra.crossproduct have an extra argument, bool max_one, which indicates that there must be no more than one match in the join. * The example module opt_sql_append is not installed in the binary packages anymore. * Added mserver5 option (--dbtrace=<path>) in order to be able to specify the output file any produced traces. * Removed function bat.setKey(). * algebra.rangejoin now has two extra arguments: anti:bit and symmetric:bit. * Added session identifier, number of workers and memory claim to the sysmon queue. * The worker (number of threads), memory (in MB) and optimizer pipeline limits can now be set per user session basis. The query and session timeouts are now set in seconds. * With required privileges an user can set resource limits for a session. * There are now versions of group.(sub)group(done) that produce a single output containing just the groups. * algebra.join and algebra.leftjoin now have forms which return a single column. The column that is returned is the left column of the two column version. * The MAL profiler now assigns the SQL TRACE output to the client record thereby avoiding the interaction with other queries, but loosing insight of competing queries. The stethoscope should be used for that. SELinux Support * There was a problem with the MonetDB SELinux support on Fedora 32. That is fixed in this release. In order to do a proper upgrade of the package if you have already installed MonetDB-selinux on Fedora 32, you may need to uninstall (dnf remove) the old package and then install the new. Whole System * A new system to deal with debug output has been implemented. There is now an option --dbtrace to mserver5 that takes a file argument to which debug output is written. The default value is the file mdbtrace.log inside the database directory. This option can also be set through the monetdb program. * The home directory of the automatically created monetdb user was changed from /var/MonetDB to /var/lib/monetdb (RPM based systems only). This home directory is (currently) not used for anything, though. * Python 2 support has been removed. There is now only support for using Python 3. * Removed support for bam and sam files. * Added mserver5 option (--set raw_strings=true|false) and monetdb database property (raw_strings=yes|no) to control interpretation of strings. Merovingian * On Fedora and RHEL systems (not RHEL 6), if monetdbd runs under systemd, when the package is updated, monetdbd (and hence any mserver5 process it runs) is restarted. * Added dbtrace mserver5 option to the daemon in order to set mserver5's output directory for the produced traces. Client Package * The monetdb-client-tools (Debian/Ubuntu) and MonetDB-client-tools (Fedora/RH) containing the stethoscope, tachograph, and tomograph has been removed. A completely new version of stethoscope will be released to replace the old version. * Removed the possibility of using the MD5 checksum for authentication purposes. It was never actively used but was there as an option. Now the option has been removed. * Add port and host as fields in the .monetdb file. * Added 'sessionid' column to system function sys.queue(), so each query gets tagged with the current session identifier * Allow monetdb user to control session and query time out and selectively stopping a client sessions with a soft termination request. ODBC Driver * The NUMERIC and FLOAT types are now handled fully. Before only DECIMAL, FLOAT, and DOUBLE were handled fully. * Some bugs were fixed in the passing back and forth between application and server of values of type GUID (UUID). MonetDB Common * The functions BATintersect, BATsemijoin, and BATsubcross have an extra argument, bool max_one, which indicates that there must be no more than one match in the join. * The "unique" property on BATs was removed. The property indicated that all values in a BAT *had* to be distinct, but this was not actually used. * A new type of candidate list has been introduced. Candidate lists are used internally to specify which rows of a column participate in an operation. Before, candidate lists always contained a list of candidate row IDs. The new candidate list type specifies a list of row IDs that should NOT be considered (negative candidates). * The maximum number of BATs in the system has been increased for 64 bit architectures. * The hash tables used internally by the system now uses a technique based on Linear Hashing which allows them to grow gracefully. This means that hash tables aren't removed and recreated nearly as often anymore. This also meant that the hash table had to be split into two files, which means that after an upgrade the hash tables have to be recreated. * Implemented a version of BATproject, called BATproject2, with two "right" arguments which conceptually follow each other. * Removed MT_mmap and MT_munmap from the list of exported functions. Use GDKmmap and GDKmunmap with the same parameters instead. * Changed the interface of the atom "fix" and "unfix" functions. They now return a value of type gdk_return to indicate success/failure. * Removed the tunique property. The tunique property indicated that all values in the column had to be distinct. It was removed because it wasn't used. * BATrangeselect now has two extra arguments: anti and symmetric (both bool). SQL Frontend * Updating the value of a sequence now requires privilege on its own schema. * The sys.querylog_enable(threshold integer) now actually enables the querylog and uses a threshold in milliseconds. * Removed UNION JOIN statements. They were dropped by the SQL:2003 standard, plus MonetDB implementation was not fully compliant. * The OFFSET value in the COPY INTO query now counts uninterpreted newlines. Before it counted "unquoted record separators" which meant that if you had a single quote on a line that you want to skip, you could not use the feature. * Added support for FROM RANGE MINVALUE TO RANGE MAXVALUE and FROM RANGE MINVALUE TO RANGE MAXVALUE WITH NULL VALUES cases in partitioned tables by range (before they weren't). * Removed support for Python 2. Python 2 itself is no longer supported. Use Python 3 instead. Functions that were declared as LANGUAGE PYTHON2 or LANGUAGE PYTHON2_MAP are changed to LANGUAGE PYTHON and LANGUAGE PYTHON_MAP respectively (without changing the Python code). * Implemented 'covar_pop' and 'covar_samp' aggregate functions, as well as their window function counterparts. Implemented 'stddev_samp', 'stddev_pop', 'var_samp', 'var_pop', 'corr' and 'group_concat' window function correspondents. * Extended SQL catalog with CREATE WINDOW syntax for user-defined SQL:2003 window functions. At the moment, window functions must be defined on the backend engine, i.e. on this case MAL. In the current implementation, the backend code generation creates two additional columns of type lng with the start and end offsets for each row. * Added prepared_statements_args view, which details the arguments for the prepared statements created in the current session. * Added sys.prepared_statements view, which lists the available prepared statements in the current session. * Added deallocate statements with the syntax 'DEALLOCATE [PREPARE] { number | ALL }', to close an existing prepared statement or all, through the SQL layer. Previously this feature was available via MAPI exclusively with the "release" command. * Added support for raw strings using the syntax r'' or R''. This means that C-like escapes will remain uninterpreted within those strings. For instance SELECT r'\"' returns a string of length two. The user needs to escape single quotes by doubling them: SELECT r''''. * Implemented ROLLUP, CUBE and GROUPING SETS from SQL:1999. They define grouping subsets used with the GROUP BY clause in order to compute partial groupings. Also, the GROUPING aggregate was added. This aggregate is a bitmask identifying the grouping columns not present in the generated grouping row when used with the operators described above. * Updated user session procedures by adding the possibility to set properties based on a session identifier. Optimizer pipeline: sys.setoptimizer(int, string) Number of worker threads: sys.setworkerlimit(int, int) Memory limits (in MB): sys.setmemorylimit(int, int) Query timeout (in ms): sys.setquerytimeout(int, int) Session timeout (in ms): sys.setsessiontimeout(int, int) The first argument corresponds to the id of the session to modify, and these procedures are bound to the monetdb user exclusively. The versions of the mentioned procedures with just the second argument were added as well, where the changes are reflected in the current user session, and therefore every user can call them. * The procedures sys.settimeout(bigint), sys.settimeout(bigint,bigint) and sys.session(bigint) are now deprecated. Instead use sys.setquerytimeout and sys.setsessiontimeout mentioned above. * Added SQL procedure sys.hot_snapshot() which can be used to write a snapshot of the database to a tar file. For example, sys.hot_snapshot('/tmp/snapshot.tar'). If compression support is compiled in, snapshots can also be compressed ('/tmp/snapshot.tar.gz'). The tar file expands to a single directory with the same name as the database that was snapshotted. This directory can be passed directly as the --dbpath argument of mserver5 or it can be copied into an existing dbfarm and started from monetdbd. Bug Fixes * 6298: unexpectedly slow execution of SELECT length(fieldname) FROM tablename LIMIT 1 queries * 6401: Suspected memory leak in mserver5 when creating/dropping tables * 6687: Count distinct very slow and use too much the hard drive * 6723: columns aliases duplicates should not be allowed. automatic aliasing required. * 6731: Add system view to allow querying of available prepared statements and their parameters * 6732: Add SQL command to close a specific prepared statement * 6750: Executing a query on a non-existing column on a remote table crashes the remote server * 6785: function sys.isaURL(url) should have been declared as sys.isaURL(string) * 6808: reveal the alarm.sleep procedure in SQL * 6813: function not_uniques(bigint) returns error when called * 6818: usage of multiple column expressions in where-clause (f(a), f(b)) in (select a, b) causes assertion failure on mserver5 * 6821: Failed to start monetdb with embedded python * 6828: Server crashes when executing a window query with ordering by EXTRACT date * 6846: Global temporary table not accessible in other connections / sessions * 6847: A simple way of speeding up impscheck for dense canditers * 6850: Idle timestamp not set * 6851: json parser doesn't parse integers correctly * 6863: thash files not released upon drop table * 6864: (I)LIKE with multiple % doen't find matches

1 0

Re: MonetDB: Jun2020 - added support for single join, ie join and ch...
by Martin Kersten 02 May '20

02 May '20

Hoi Niels, I saw the check-in and ran the SF-100 test. The new results are: [mk@zoom 2020-05-02T21:20:46:SF-100]$ grep all timings.csv SF-100,all, 1, 263.423964 seconds SF-100,all, 2, 267.062088 seconds SF-100,all, 3, 267.019248 seconds Compared to yesterday's email: [mk@zoom 2020-04-29T16:09:48:SF-100]$ grep all timings.csv SF-100,all, 1, 263.779505 seconds SF-100,all, 2, 268.900062 seconds SF-100,all, 3, 269.732425 seconds On 04/04/2020 14:13, Martin Kersten wrote: > I ran the Jun2020 SF100 version on zoom again. > > [mk@zoom 2020-04-04T11:57:10:SF-100]$ grep all timings.csv > SF-100,all, 1, 238.894476 seconds > SF-100,all, 2, 246.591707 seconds > SF-100,all, 3, 244.333189 seconds > > average 243

1 0

Re: MonetDB: msk-type - Initial support for the msk type.
by Martin Kersten 24 Apr '20

24 Apr '20

wil je dat ik het doortrek naar MAL niveau? > On 24 Apr 2020, at 17:30, Sjoerd Mullender <commits+sjoerd=acm.org(a)monetdb.org> wrote: > > Changeset: abf00b42a524 for MonetDB > URL: https://dev.monetdb.org/hg/MonetDB?cmd=changeset;node=abf00b42a524 > Modified Files: > clients/Tests/exports.stable.out > gdk/gdk.h > gdk/gdk_atoms.c > gdk/gdk_bat.c > gdk/gdk_batop.c > gdk/gdk_calc.c > gdk/gdk_hash.c > gdk/gdk_value.c > monetdb5/mal/mal_type.c > monetdb5/modules/atoms/inet.c > monetdb5/modules/kernel/bat5.c > sql/backends/monet5/UDF/capi/capi.c > sql/backends/monet5/sql_result.c > Branch: msk-type > Log Message: > > Initial support for the msk type. > > > diffs (truncated from 1402 to 300 lines): > > diff --git a/clients/Tests/exports.stable.out b/clients/Tests/exports.stable.out > --- a/clients/Tests/exports.stable.out > +++ b/clients/Tests/exports.stable.out > @@ -1273,7 +1273,7 @@ str INETnetmask(inet *retval, const inet > str INETnetwork(inet *retval, const inet *val); > str INETnew(inet *retval, str *in); > const inet *INETnull(void); > -str INETsetmasklen(inet *retval, const inet *val, const int *msk); > +str INETsetmasklen(inet *retval, const inet *val, const int *mask); > str INETtext(str *retval, const inet *val); > ssize_t INETtoString(str *retval, size_t *len, const inet *handle, bool external); > str INSPECTatom_names(bat *ret); > diff --git a/gdk/gdk.h b/gdk/gdk.h > --- a/gdk/gdk.h > +++ b/gdk/gdk.h > @@ -425,7 +425,8 @@ > > enum { > TYPE_void = 0, > - TYPE_bit, > + TYPE_msk, /* bit mask */ > + TYPE_bit, /* TRUE, FALSE, or nil */ > TYPE_bte, > TYPE_sht, > TYPE_bat, /* BAT id: index in BBPcache */ > @@ -445,6 +446,7 @@ enum { > TYPE_any = 255, /* limit types to <255! */ > }; > > +typedef bool msk; > typedef int8_t bit; > typedef int8_t bte; > typedef int16_t sht; > @@ -602,6 +604,7 @@ typedef struct { > oid oval; > sht shval; > bte btval; > + msk mval; > flt fval; > ptr pval; > bat bval; > @@ -626,7 +629,7 @@ gdk_export void VALclear(ValPtr v); > gdk_export ValPtr VALset(ValPtr v, int t, void *p); > gdk_export void *VALget(ValPtr v); > gdk_export int VALcmp(const ValRecord *p, const ValRecord *q); > -gdk_export int VALisnil(const ValRecord *v); > +gdk_export bool VALisnil(const ValRecord *v); > > /* > * @- The BAT record > @@ -746,7 +749,10 @@ typedef struct BAT { > > typedef struct BATiter { > BAT *b; > - oid tvid; > + union { > + oid tvid; > + bool tmsk; > + }; > } BATiter; > > /* macros to hide complexity of the BAT structure */ > @@ -772,6 +778,33 @@ typedef struct BATiter { > #define tprops T.props > > > +/* some access functions for the bitmask type */ > +static inline void > +mskSet(BAT *b, BUN p) > +{ > + ((uint32_t *) b->theap.base)[p / 32] |= 1U << (p % 32); > +} > + > +static inline void > +mskClr(BAT *b, BUN p) > +{ > + ((uint32_t *) b->theap.base)[p / 32] &= ~(1U << (p % 32)); > +} > + > +static inline void > +mskSetVal(BAT *b, BUN p, msk v) > +{ > + if (v) > + mskSet(b, p); > + else > + mskClr(b, p); > +} > + > +static inline msk > +mskGet(BAT *b, BUN p) > +{ > + return ((uint32_t *) b->theap.base)[p / 32] & (1U << (p % 32)); > +} > > /* > * @- Heap Management > @@ -925,7 +958,11 @@ gdk_export BUN BUNfnd(BAT *b, const void > > #define Tsize(b) ((b)->twidth) > > -#define tailsize(b,p) ((b)->ttype?((size_t)(p))<<(b)->tshift:0) > +#define tailsize(b,p) ((b)->ttype ? \ > + (ATOMstorage((b)->ttype) == TYPE_msk ? \ > + (((size_t) (p) + 31) / 32) * 4 : \ > + ((size_t) (p)) << (b)->tshift) : \ > + 0) > > #define Tloc(b,p) ((void *)((b)->theap.base+(((size_t)(p))<<(b)->tshift))) > > @@ -935,7 +972,7 @@ typedef var_t stridx_t; > > #define BUNtvaroff(bi,p) VarHeapVal((bi).b->theap.base, (p), (bi).b->twidth) > > -#define BUNtloc(bi,p) Tloc((bi).b,p) > +#define BUNtloc(bi,p) (ATOMstorage((bi).b->ttype) == TYPE_msk ? Tmsk(&(bi), p) : Tloc((bi).b,p)) > #define BUNtpos(bi,p) Tpos(&(bi),p) > #define BUNtvar(bi,p) (assert((bi).b->ttype && (bi).b->tvarsized), (void *) (Tbase((bi).b)+BUNtvaroff(bi,p))) > #define BUNtail(bi,p) ((bi).b->ttype?(bi).b->tvarsized?BUNtvar(bi,p):BUNtloc(bi,p):BUNtpos(bi,p)) > @@ -1519,6 +1556,8 @@ Tputvalue(BAT *b, BUN p, const void *v, > break; > #endif > } > + } else if (b->ttype == TYPE_msk) { > + mskSetVal(b, p, * (msk *) v); > } else { > return ATOMputFIX(b->ttype, Tloc(b, p), v); > } > @@ -1528,7 +1567,11 @@ Tputvalue(BAT *b, BUN p, const void *v, > static inline gdk_return __attribute__((__warn_unused_result__)) > tfastins_nocheck(BAT *b, BUN p, const void *v, int s) > { > - b->theap.free += s; > + if (ATOMstorage(b->ttype) == TYPE_msk) { > + if (p % 32 == 0) > + b->theap.free += 4; > + } else > + b->theap.free += s; > return Tputvalue(b, p, v, false); > } > > @@ -1676,6 +1719,7 @@ VALptr(const ValRecord *v) > { > switch (ATOMstorage(v->vtype)) { > case TYPE_void: return (const void *) &v->val.oval; > + case TYPE_msk: return (const void *) &v->val.mval; > case TYPE_bte: return (const void *) &v->val.btval; > case TYPE_sht: return (const void *) &v->val.shval; > case TYPE_int: return (const void *) &v->val.ival; > @@ -1734,8 +1778,6 @@ gdk_export void *THRdata[THREADDATA]; > #define THRget_errbuf(t) ((char*)t->data[2]) > #define THRset_errbuf(t,b) (t->data[2] = b) > > -#ifndef GDK_NOLINK > - > static inline bat > BBPcheck(bat x, const char *y) > { > @@ -1773,7 +1815,12 @@ Tpos(BATiter *bi, BUN p) > return (void*)&bi->tvid; > } > > -#endif > +static inline void * > +Tmsk(BATiter *bi, BUN p) > +{ > + bi->tmsk = mskGet(bi->b, p); > + return &bi->tmsk; > +} > > /* > * @+ Transaction Management > diff --git a/gdk/gdk_atoms.c b/gdk/gdk_atoms.c > --- a/gdk/gdk_atoms.c > +++ b/gdk/gdk_atoms.c > @@ -36,6 +36,12 @@ > * NaN). */ > > static int > +mskCmp(const msk *l, const msk *r) > +{ > + return (*l > *r) - (*l < *r); > +} > + > +static int > bteCmp(const bte *l, const bte *r) > { > return (*l > *r) - (*l < *r); > @@ -450,6 +456,42 @@ voidWrite(const void *a, stream *s, size > * experiments showed that library function is even slightly faster and we > * now also support True/False (and trUe/FAlSE should this become a thing). > */ > +static ssize_t > +mskFromStr(const char *src, size_t *len, msk **dst, bool external) > +{ > + const char *p = src; > + > + (void) external; > + atommem(sizeof(msk)); > + > + if (strNil(src)) > + return -1; > + > + while (GDKisspace(*p)) > + p++; > + if (*p == '0') { > + **dst = 0; > + p++; > + } else if (*p == '1') { > + **dst = 1; > + p++; > + } else { > + return -1; > + } > + while (GDKisspace(*p)) > + p++; > + return (ssize_t) (p - src); > +} > + > +static ssize_t > +mskToStr(char **dst, size_t *len, const msk *src, bool external) > +{ > + (void) external; > + atommem(2); > + strcpy(*dst, *src ? "1" : "0"); > + return 1; > +} > + > ssize_t > bitFromStr(const char *src, size_t *len, bit **dst, bool external) > { > @@ -1202,6 +1244,17 @@ atomDesc BATatoms[MAXATOMS] = { > .atomCmp = (int (*)(const void *, const void *)) bteCmp, > .atomHash = (BUN (*)(const void *)) bteHash, > }, > + [TYPE_msk] = { > + .name = "msk", > + .storage = TYPE_msk, > + .linear = false, > + .size = 1, /* really 1/8 */ > + .atomFromStr = (ssize_t (*)(const char *, size_t *, void **, bool)) mskFromStr, > + .atomToStr = (ssize_t (*)(char **, size_t *, const void *, bool)) mskToStr, > +// .atomRead = (void *(*)(void *, stream *, size_t)) mskRead, > +// .atomWrite = (gdk_return (*)(const void *, stream *, size_t)) mskWrite, > + .atomCmp = (int (*)(const void *, const void *)) mskCmp, > + }, > [TYPE_bte] = { > .name = "bte", > .storage = TYPE_bte, > diff --git a/gdk/gdk_bat.c b/gdk/gdk_bat.c > --- a/gdk/gdk_bat.c > +++ b/gdk/gdk_bat.c > @@ -184,7 +184,9 @@ COLnew(oid hseq, int tt, BUN cap, role_t > /* round up to multiple of BATTINY */ > if (cap < BUN_MAX - BATTINY) > cap = (cap + BATTINY - 1) & ~(BATTINY - 1); > - if (cap < BATTINY) > + if (ATOMstorage(tt) == TYPE_msk && cap < 8*BATTINY) > + cap = 8*BATTINY; > + else if (cap < BATTINY) > cap = BATTINY; > /* limit the size */ > if (cap > BUN_MAX) > @@ -197,6 +199,9 @@ COLnew(oid hseq, int tt, BUN cap, role_t > BATsetdims(bn); > bn->batCapacity = cap; > > + if (ATOMstorage(tt) == TYPE_msk) > + cap /= 8; /* 8 values per byte */ > + > /* alloc the main heaps */ > if (tt && HEAPalloc(&bn->theap, cap, bn->twidth) != GDK_SUCCEED) { > goto bailout; > @@ -339,12 +344,16 @@ BATattach(int tt, const char *heapfile, > GDKerror("heapfile size not integral number of atoms\n"); > return NULL; > } > - if ((size_t) (st.st_size / atomsize) > (size_t) BUN_MAX) { > + if (ATOMstorage(tt) == TYPE_msk ? > + (st.st_size > (off_t) (BUN_MAX / 8)) : > + ((size_t) (st.st_size / atomsize) > (size_t) BUN_MAX)) { > fclose(f); > GDKerror("heapfile too large\n"); > return NULL; > } > - cap = (BUN) (st.st_size / atomsize); > + cap = (BUN) (ATOMstorage(tt) == TYPE_msk ? > + st.st_size * 8 : > + st.st_size / atomsize); > bn = COLnew(0, tt, cap, role); > if (bn == NULL) { > fclose(f); > @@ -371,8 +380,8 @@ BATattach(int tt, const char *heapfile, > bn->trevsorted = false; > bn->tkey = false; > } else { > - bn->tsorted = true; > - bn->trevsorted = true; > + bn->tsorted = ATOMlinear(tt); > + bn->trevsorted = ATOMlinear(tt); > bn->tkey = true; > _______________________________________________ > checkin-list mailing list > checkin-list(a)monetdb.org > https://www.monetdb.org/mailman/listinfo/checkin-list

2 1

Re: MonetDB: msk-type - Initial support for the msk type.
by Martin Kersten 24 Apr '20

24 Apr '20

wil je dat ik het doortrek naar MAL niveau? > On 24 Apr 2020, at 17:30, Sjoerd Mullender <commits+sjoerd=acm.org(a)monetdb.org> wrote: > > Changeset: abf00b42a524 for MonetDB > URL: https://dev.monetdb.org/hg/MonetDB?cmd=changeset;node=abf00b42a524 > Modified Files: > clients/Tests/exports.stable.out > gdk/gdk.h > gdk/gdk_atoms.c > gdk/gdk_bat.c > gdk/gdk_batop.c > gdk/gdk_calc.c > gdk/gdk_hash.c > gdk/gdk_value.c > monetdb5/mal/mal_type.c > monetdb5/modules/atoms/inet.c > monetdb5/modules/kernel/bat5.c > sql/backends/monet5/UDF/capi/capi.c > sql/backends/monet5/sql_result.c > Branch: msk-type > Log Message: > > Initial support for the msk type. > > > diffs (truncated from 1402 to 300 lines): > > diff --git a/clients/Tests/exports.stable.out b/clients/Tests/exports.stable.out > --- a/clients/Tests/exports.stable.out > +++ b/clients/Tests/exports.stable.out > @@ -1273,7 +1273,7 @@ str INETnetmask(inet *retval, const inet > str INETnetwork(inet *retval, const inet *val); > str INETnew(inet *retval, str *in); > const inet *INETnull(void); > -str INETsetmasklen(inet *retval, const inet *val, const int *msk); > +str INETsetmasklen(inet *retval, const inet *val, const int *mask); > str INETtext(str *retval, const inet *val); > ssize_t INETtoString(str *retval, size_t *len, const inet *handle, bool external); > str INSPECTatom_names(bat *ret); > diff --git a/gdk/gdk.h b/gdk/gdk.h > --- a/gdk/gdk.h > +++ b/gdk/gdk.h > @@ -425,7 +425,8 @@ > > enum { > TYPE_void = 0, > - TYPE_bit, > + TYPE_msk, /* bit mask */ > + TYPE_bit, /* TRUE, FALSE, or nil */ > TYPE_bte, > TYPE_sht, > TYPE_bat, /* BAT id: index in BBPcache */ > @@ -445,6 +446,7 @@ enum { > TYPE_any = 255, /* limit types to <255! */ > }; > > +typedef bool msk; > typedef int8_t bit; > typedef int8_t bte; > typedef int16_t sht; > @@ -602,6 +604,7 @@ typedef struct { > oid oval; > sht shval; > bte btval; > + msk mval; > flt fval; > ptr pval; > bat bval; > @@ -626,7 +629,7 @@ gdk_export void VALclear(ValPtr v); > gdk_export ValPtr VALset(ValPtr v, int t, void *p); > gdk_export void *VALget(ValPtr v); > gdk_export int VALcmp(const ValRecord *p, const ValRecord *q); > -gdk_export int VALisnil(const ValRecord *v); > +gdk_export bool VALisnil(const ValRecord *v); > > /* > * @- The BAT record > @@ -746,7 +749,10 @@ typedef struct BAT { > > typedef struct BATiter { > BAT *b; > - oid tvid; > + union { > + oid tvid; > + bool tmsk; > + }; > } BATiter; > > /* macros to hide complexity of the BAT structure */ > @@ -772,6 +778,33 @@ typedef struct BATiter { > #define tprops T.props > > > +/* some access functions for the bitmask type */ > +static inline void > +mskSet(BAT *b, BUN p) > +{ > + ((uint32_t *) b->theap.base)[p / 32] |= 1U << (p % 32); > +} > + > +static inline void > +mskClr(BAT *b, BUN p) > +{ > + ((uint32_t *) b->theap.base)[p / 32] &= ~(1U << (p % 32)); > +} > + > +static inline void > +mskSetVal(BAT *b, BUN p, msk v) > +{ > + if (v) > + mskSet(b, p); > + else > + mskClr(b, p); > +} > + > +static inline msk > +mskGet(BAT *b, BUN p) > +{ > + return ((uint32_t *) b->theap.base)[p / 32] & (1U << (p % 32)); > +} > > /* > * @- Heap Management > @@ -925,7 +958,11 @@ gdk_export BUN BUNfnd(BAT *b, const void > > #define Tsize(b) ((b)->twidth) > > -#define tailsize(b,p) ((b)->ttype?((size_t)(p))<<(b)->tshift:0) > +#define tailsize(b,p) ((b)->ttype ? \ > + (ATOMstorage((b)->ttype) == TYPE_msk ? \ > + (((size_t) (p) + 31) / 32) * 4 : \ > + ((size_t) (p)) << (b)->tshift) : \ > + 0) > > #define Tloc(b,p) ((void *)((b)->theap.base+(((size_t)(p))<<(b)->tshift))) > > @@ -935,7 +972,7 @@ typedef var_t stridx_t; > > #define BUNtvaroff(bi,p) VarHeapVal((bi).b->theap.base, (p), (bi).b->twidth) > > -#define BUNtloc(bi,p) Tloc((bi).b,p) > +#define BUNtloc(bi,p) (ATOMstorage((bi).b->ttype) == TYPE_msk ? Tmsk(&(bi), p) : Tloc((bi).b,p)) > #define BUNtpos(bi,p) Tpos(&(bi),p) > #define BUNtvar(bi,p) (assert((bi).b->ttype && (bi).b->tvarsized), (void *) (Tbase((bi).b)+BUNtvaroff(bi,p))) > #define BUNtail(bi,p) ((bi).b->ttype?(bi).b->tvarsized?BUNtvar(bi,p):BUNtloc(bi,p):BUNtpos(bi,p)) > @@ -1519,6 +1556,8 @@ Tputvalue(BAT *b, BUN p, const void *v, > break; > #endif > } > + } else if (b->ttype == TYPE_msk) { > + mskSetVal(b, p, * (msk *) v); > } else { > return ATOMputFIX(b->ttype, Tloc(b, p), v); > } > @@ -1528,7 +1567,11 @@ Tputvalue(BAT *b, BUN p, const void *v, > static inline gdk_return __attribute__((__warn_unused_result__)) > tfastins_nocheck(BAT *b, BUN p, const void *v, int s) > { > - b->theap.free += s; > + if (ATOMstorage(b->ttype) == TYPE_msk) { > + if (p % 32 == 0) > + b->theap.free += 4; > + } else > + b->theap.free += s; > return Tputvalue(b, p, v, false); > } > > @@ -1676,6 +1719,7 @@ VALptr(const ValRecord *v) > { > switch (ATOMstorage(v->vtype)) { > case TYPE_void: return (const void *) &v->val.oval; > + case TYPE_msk: return (const void *) &v->val.mval; > case TYPE_bte: return (const void *) &v->val.btval; > case TYPE_sht: return (const void *) &v->val.shval; > case TYPE_int: return (const void *) &v->val.ival; > @@ -1734,8 +1778,6 @@ gdk_export void *THRdata[THREADDATA]; > #define THRget_errbuf(t) ((char*)t->data[2]) > #define THRset_errbuf(t,b) (t->data[2] = b) > > -#ifndef GDK_NOLINK > - > static inline bat > BBPcheck(bat x, const char *y) > { > @@ -1773,7 +1815,12 @@ Tpos(BATiter *bi, BUN p) > return (void*)&bi->tvid; > } > > -#endif > +static inline void * > +Tmsk(BATiter *bi, BUN p) > +{ > + bi->tmsk = mskGet(bi->b, p); > + return &bi->tmsk; > +} > > /* > * @+ Transaction Management > diff --git a/gdk/gdk_atoms.c b/gdk/gdk_atoms.c > --- a/gdk/gdk_atoms.c > +++ b/gdk/gdk_atoms.c > @@ -36,6 +36,12 @@ > * NaN). */ > > static int > +mskCmp(const msk *l, const msk *r) > +{ > + return (*l > *r) - (*l < *r); > +} > + > +static int > bteCmp(const bte *l, const bte *r) > { > return (*l > *r) - (*l < *r); > @@ -450,6 +456,42 @@ voidWrite(const void *a, stream *s, size > * experiments showed that library function is even slightly faster and we > * now also support True/False (and trUe/FAlSE should this become a thing). > */ > +static ssize_t > +mskFromStr(const char *src, size_t *len, msk **dst, bool external) > +{ > + const char *p = src; > + > + (void) external; > + atommem(sizeof(msk)); > + > + if (strNil(src)) > + return -1; > + > + while (GDKisspace(*p)) > + p++; > + if (*p == '0') { > + **dst = 0; > + p++; > + } else if (*p == '1') { > + **dst = 1; > + p++; > + } else { > + return -1; > + } > + while (GDKisspace(*p)) > + p++; > + return (ssize_t) (p - src); > +} > + > +static ssize_t > +mskToStr(char **dst, size_t *len, const msk *src, bool external) > +{ > + (void) external; > + atommem(2); > + strcpy(*dst, *src ? "1" : "0"); > + return 1; > +} > + > ssize_t > bitFromStr(const char *src, size_t *len, bit **dst, bool external) > { > @@ -1202,6 +1244,17 @@ atomDesc BATatoms[MAXATOMS] = { > .atomCmp = (int (*)(const void *, const void *)) bteCmp, > .atomHash = (BUN (*)(const void *)) bteHash, > }, > + [TYPE_msk] = { > + .name = "msk", > + .storage = TYPE_msk, > + .linear = false, > + .size = 1, /* really 1/8 */ > + .atomFromStr = (ssize_t (*)(const char *, size_t *, void **, bool)) mskFromStr, > + .atomToStr = (ssize_t (*)(char **, size_t *, const void *, bool)) mskToStr, > +// .atomRead = (void *(*)(void *, stream *, size_t)) mskRead, > +// .atomWrite = (gdk_return (*)(const void *, stream *, size_t)) mskWrite, > + .atomCmp = (int (*)(const void *, const void *)) mskCmp, > + }, > [TYPE_bte] = { > .name = "bte", > .storage = TYPE_bte, > diff --git a/gdk/gdk_bat.c b/gdk/gdk_bat.c > --- a/gdk/gdk_bat.c > +++ b/gdk/gdk_bat.c > @@ -184,7 +184,9 @@ COLnew(oid hseq, int tt, BUN cap, role_t > /* round up to multiple of BATTINY */ > if (cap < BUN_MAX - BATTINY) > cap = (cap + BATTINY - 1) & ~(BATTINY - 1); > - if (cap < BATTINY) > + if (ATOMstorage(tt) == TYPE_msk && cap < 8*BATTINY) > + cap = 8*BATTINY; > + else if (cap < BATTINY) > cap = BATTINY; > /* limit the size */ > if (cap > BUN_MAX) > @@ -197,6 +199,9 @@ COLnew(oid hseq, int tt, BUN cap, role_t > BATsetdims(bn); > bn->batCapacity = cap; > > + if (ATOMstorage(tt) == TYPE_msk) > + cap /= 8; /* 8 values per byte */ > + > /* alloc the main heaps */ > if (tt && HEAPalloc(&bn->theap, cap, bn->twidth) != GDK_SUCCEED) { > goto bailout; > @@ -339,12 +344,16 @@ BATattach(int tt, const char *heapfile, > GDKerror("heapfile size not integral number of atoms\n"); > return NULL; > } > - if ((size_t) (st.st_size / atomsize) > (size_t) BUN_MAX) { > + if (ATOMstorage(tt) == TYPE_msk ? > + (st.st_size > (off_t) (BUN_MAX / 8)) : > + ((size_t) (st.st_size / atomsize) > (size_t) BUN_MAX)) { > fclose(f); > GDKerror("heapfile too large\n"); > return NULL; > } > - cap = (BUN) (st.st_size / atomsize); > + cap = (BUN) (ATOMstorage(tt) == TYPE_msk ? > + st.st_size * 8 : > + st.st_size / atomsize); > bn = COLnew(0, tt, cap, role); > if (bn == NULL) { > fclose(f); > @@ -371,8 +380,8 @@ BATattach(int tt, const char *heapfile, > bn->trevsorted = false; > bn->tkey = false; > } else { > - bn->tsorted = true; > - bn->trevsorted = true; > + bn->tsorted = ATOMlinear(tt); > + bn->trevsorted = ATOMlinear(tt); > bn->tkey = true; > _______________________________________________ > checkin-list mailing list > checkin-list(a)monetdb.org > https://www.monetdb.org/mailman/listinfo/checkin-list

1 0

New Jun2020 stable branch created
by Sjoerd Mullender 18 Mar '20

18 Mar '20

I have created a new stable branch for the next feature release. This branch was created in order to stabilize the code before the feature release. >From now on *only* bug fixes may go to this new stable branch. I reserve the right to undo any changes that do not comply with this rule. I also reserve the right to admit changes that are not strictly bug fixes. The name for the branch is Jun2020. So in each of your stable checkout directories you can do hg pull hg update -rJun2020 to update to this new branch. -- Sjoerd Mullender

1 0

Announcement: New Nov2019-SP2 Bugfix release of MonetDB suite
by Sjoerd Mullender 21 Feb '20

21 Feb '20

The MonetDB team at MonetDB BV is pleased to announce the Nov2019-SP2 bugfix release of the MonetDB suite of programs. More information about MonetDB can be found on our website at <https://www.monetdb.org/>. For details on this release, please see the release notes at <https://www.monetdb.org/Downloads/ReleaseNotes>. As usual, the download location is <https://www.monetdb.org/downloads/>. Nov 2019-SP2 bugfix release (11.35.15) Bug Fixes * 6805: Using the cascade operator in a drop table statement ends in an exit from the Monetdb shell. * 6807: Median_avg and quantile_avg ignore NULL values * 6815: query with ifthenelse() crashes mserver5 * 6816: Monetdb Crashes on INSERT statement after ALTER statement in another connection * 6817: running analyze on a schema which contains a stream table stops with an error * 6819: functions do not persist

1 0

Re: MonetDB: gdk_tracer - small compilation fixes for windows
by Martin Kersten 27 Dec '19

27 Dec '19

Niels, A big check-in is coming. I went over the wlcr code and 'removed' the background thread. Now only rolling forward using wlr.replicate(limit) or (exclusively) the background thread using a beat. Test wlr70 still fails, because the code in wlr.c line 520 is still wrong (call wlr.replicate(now())) strftime(timelimit, sizeof(timelimit), "%Y-%m-%d %H:%M:%S.000", (void*) getArgReference(stk, pci, 1)); advice?

1 0

Announcement: New Nov2019-SP1 Bugfix release of MonetDB suite
by Sjoerd Mullender 20 Dec '19

20 Dec '19

The MonetDB team at CWI/MonetDB BV is pleased to announce the Nov2019-SP1 bugfix release of the MonetDB suite of programs. More information about MonetDB can be found on our website at <https://www.monetdb.org/>. For details on this release, please see the release notes at <https://www.monetdb.org/Downloads/ReleaseNotes>. As usual, the download location is <https://dev.monetdb.org/downloads/>. Nov 2019-SP1 bugfix release (11.35.9) Bug Fixes * 6723: columns aliases duplicates should not be allowed. automatic aliasing required. * 6724: Prepare confuses types when more than one argument is used * 6726: Python aggregation does not create aggr_group when aggregating over all rows * 6765: GRANT SELECT privilege on a subset of table columns results in access denied error when selecting the same columns from the table * 6790: Count distinct giving wrong results * 6791: str_to_time('11:40', '%H:%M') creates wrong time value * 6792: JSON path compiler accepts invalid input * 6793: cast(interval second value to int or decimal) is wrong (by a factor of 1000), cast(interval month value to decimal or floating point) fails * 6794: external name fits.listdir not bound (sys.listdir) Fatal error during initialization: * 6796: Incorrect crash time reported by monetdb tool after crash of mserver5 * 6798: json.text off by one error * 6804: DNS resolution of 0.0.0.0 fails on recent Ubuntus

1 0

Announcement: New Nov2019 Feature release of MonetDB suite
by Sjoerd Mullender 28 Nov '19

28 Nov '19

The MonetDB team at CWI/MonetDB BV is pleased to announce the Nov2019 feature release of the MonetDB suite of programs. More information about MonetDB can be found on our website at <https://www.monetdb.org/>. For details on this release, please see the release notes at <https://www.monetdb.org/Downloads/ReleaseNotes>. As usual, the download location is <https://dev.monetdb.org/downloads/>. Nov 2019 feature release (11.35.3) MonetDB5 Server * The server "console" has been removed, as has the --daemon option. The server now doesn't read from standard input anymore. The way to stop a server is by sending it a TERM signal (on Linux/Unix) or by sending it an interrupt signal (usually control-C -- on all systems). * Implemented a function bat.diffcand to calculate difference of two candidate lists. * The mtime module was completely rewritten, the atom types date, daytime, and timestamp were changed. Upgrade code for BATs containing these types has been implemented. The old daytime type used a 32 bit integer to record milliseconds since the start of the day. The new daytime type uses a 64 bit integer to record microseconds since the start of the day. The old date type recorded days since or before the year 1. The new daytime type records the day of the month and the number of months since the year -4712 separately in a single 32 bit integer of which only 26 bits are used. Dates now use the proleptic Gregorian calendar, meaning the normal Gregorian callendar has been extended backward, and before the year 1, we have the year 0. Both the old and new timestamp types are a combination of a daytime and a date value, but since those types have changed, the timestamp type has also changed. The new date type has a smaller range than the old. The new date range is from the year -4712 to the year 170049. During conversion of date and timestamp columns, the dates are clamped to this range. * The tzone and rule atom types have been removed. They were not used by any code, and they were defined in a non-portable way. * Added "mapi_ipv6" property to monet_options to force ipv6 address binding only. This property is inherited while forking from monetdbd if it is also set there. * Removed (bat)calc.between_symmetric and changed (bat)calc.between by adding a number of extra arguments, all of type :bit: symmetric, low inclusive, high inclusive, nils false. Merovingian * Added "vmmaxsize" and "memmaxsize" mserver5 options to the daemon in order to set mserver5's maximum virtual and committed memory respectively. * Added ipv6 property to monetdbd properties to force IPv6 addresses binding only. By default this property is false to allow IPv4 addresses as well. MonetDB Common * BATcalcbetween and all its variants now have an extra bool parameter "anti" to invert the test. * All forms of BATcalcbetween and VARcalcbetween have extra bool arguments for low inclusive, high inclusive and nils false. The latter causes the result to be false instead of nil if the value being checked is nil. SQL Frontend * Removed functions json.text(string) returns string and json.text(int) returns string. Their MAL implementation didn't exist, so they were meaningless. * There are new aggregate functions sys.median_avg and sys.quantile_avg that return the interpolated value if the median/quantile doesn't fall exactly on a particular row. These functions always return a value of type DOUBLE and only work for numeric types (various width integers and floating point). * Added sys.deltas ("schema" string, "table" string, "column" string) returns table ("values" bigint) system function which returns a single column with 6 values: a flag indicating if the column's upper table is cleared or not, the count of the RDONLY, RD_INS and RD_UPD_ID deltas of the column itself, the number of deleted values of the column's table, as well as the level of the current transaction in the transaction level tree. It should be used for debugging purposes only. * Added "VALUES row_list" statement as a top SQL projection statement. * The implementation of in-expression now follows a join-based approach instead of using iterative union/selects. This greatly improves performance for large in-value-lists. Furthermore the old approach has large in-value-lists generate large MAL plans. This is now no longer the case. * Strings are now limited to 1GB, double-quoted tokens are limited to 2kB. These sizes are bytes of (UTF-8 encoded) input data. Bug Fixes * 3533: SQL aggregate functions avg(), sum() and median() return an error when used on a column with datatype interval second * 6134: Query produces error: HEAPalloc: Insufficient space for HEAP of 1168033427456 bytes. * 6613: LATERAL crash /.../rel_bin.c:1473: rel2bin_table: Assertion `0' failed. * 6683: Bug in subselect * 6686: Bug in subselect (count function) * 6688: Bug in subselect (or condition) * 6689: Trying to improve the performance of SQL queries with a large list of members in IN clause. * 6695: timestamp transformation * 6700: Monetdb Bugs in Subselect statements: * 6722: window functions issues * 6740: while upgrading the database from verison (MonetDB-11.27.13) to (MonetDB-11.33.3) we are unable to bring up the database * 6754: in mclient a strang msg is reported after issueing command: set schema sys; * 6755: Assertion failure in rel_bin.c * 6756: Error in optimizer garbageCollector on merge tables select * 6757: Double free or corruption (out) * 6758: SIGSEGV in __strcmp_sse2_unaligned() * 6759: Python JSON loader creates invalid data type for strings * 6761: Error: Program contains errors.:(NONE).multiplex * 6762: mserver5 crashes on (re-)start * 6764: mserver5 crashes with corruption, double free, invalid size or invalid pointer * 6766: Missing bulk implementation for get_value and next_value calls * 6769: ProfilerStart is not threadsafe * 6771: R-devel * 6772: TRACE includes information from all active sessions * 6773: json.filter returns corrupted string when selecting JSON null value * 6774: PROD aggregation gives wrong result * 6775: NOT IN with an AND containing an OR gives wrong result * 6776: Creating a table with a full outer join query gives type with wrong digits on the joined key. * 6779: Using Windows Messages translation for errno error codes. * 6780: Wrong value of the rank function * 6781: Insert after index creation crash * 6783: AVG changes scale of its results * 6784: function sys.isauuid(string) should return false if string value cannot be converted to a UUID

1 0

Re: MonetDB: default - Compilation fixes
by Martin Kersten 08 Nov '19

08 Nov '19

On 08/11/2019 09:48, Pedro Ferreira wrote: > + ATOMIC_TYPE workers; /* Actual number of concurrent workers */ This is strange!! They were removed and turned into normal variables. Now I wonder if the correct merge has taken place. I will check the code

1 0

JOIN using OR and winerror translation
by Daniel Zvinca 22 Oct '19

22 Oct '19

Hello, During a complex join between two tables that involve an OR operation, I received the following error message: !OS: The printer is out of paper. This corresponds to error number 28 in Windows OS, yet in errno.h, 28 is for ENOSPC, which is the expected one. In the meanwhile, I fixed that join, replacing the complex condition that included one OR into a UNION ALL (conditions made that possible), but I think it's worth investigating possible optimization that would do this internally (transforming join conditions using OR with plain UNION). Anyway, if above suggestion is just something to think about (as long as it always exists a solution in the hands of the SQL developer), the winerror translation certainly requires a review. (mutils.c / int winerror(int e)); Thank you, Dan

2 1

R-devel
by Arjen P. de Vries 09 Oct '19

09 Oct '19

Hi, Just installing from source for the first time since quite a while :-) On a FC30 system, using: ./configure --disable-debug --disable-developer --disable-assert --enable-optimize the bootstrap / configure steps did not complain about R-devel missing, but then issuing make failed on not finding the Rembedded.h include. Maybe the configure script needs a patch! All fine after installing R-devel, thanks for great docs! Cheers, Arjen -- ==================================================================== ICIS, office M1.02.07 Radboud University Mercator 1 Faculty of Science Toernooiveld 212 arjen(a)cs.ru.nl NL-6525 EC Nijmegen, The Netherlands +31-(0)24-365 2354 ===================== http://www.informagus.nl/ ====================

2 1

case performance
by Daniel Zvinca 08 Oct '19

08 Oct '19

Hello, I am exploring MonetDB queries performance that are generated based on user parameters/preferences. To be more specific the where clause is generated based on a user interface mapping user input parameters straight into SQL statements. See below one example of this transformation. FIELD contains PATTERN (case sensitive/insensitive) is transformed into select ... from TABLE where (case when (user_case_sensitive_pref=1) then ("FIELD" like '%PATTERN%') else ("FIELD" ilike '%PATTERN%') end); Performance of this in comparison with query generated from within the code: char * query; if ( user_case_sensitive_pref ) query = "select ... from TABLE where \"FIELD\" like '%PATTERN%'"; else query = "select ... from TABLE where \"FIELD\" ilike '%PATTERN%'"; query_execute( query); is huge (second approach takes 0.12 sec, vs 6 sec for first one). But it requires a far more complex implementation. First I thought this has to do with user variables but even if the statement is select ... from TABLE where (case when TRUE then ("FIELD" like '%PATTERN%') else ("FIELD" ilike '%PATTERN%') end); timing is similarly slow. This narrows down to "case" performance issue, as far as I can see. Am I wrong? Thank you, Dan

3 8

Announcement: New Apr2019 Feature release of MonetDB suite
by Sjoerd Mullender 26 Sep '19

26 Sep '19

The MonetDB team at CWI/MonetDB BV is pleased to announce the Apr2019 feature release of the MonetDB suite of programs. More information about MonetDB can be found on our website at <https://www.monetdb.org/>. For details on this release, please see the release notes at <https://www.monetdb.org/Downloads/ReleaseNotes>. As usual, the download location is <https://dev.monetdb.org/downloads/>. Apr 2019 feature release (11.33.3) MonetDB5 Server * Changed the way blobs are ordered. Before, shorter blobs came before longer, now the contents is compared first. * Removed function blob.tostring() since it cannot guarantee that the resulting string is properly encoded in UTF-8. * Removed the type "sqlblob", changed the external format of the type "blob" to be like what "sqlblob" used to be. In other words, the "blob" type is now SQL-compatible. * The MAL functions algebra.sort and algebra.firstn now have a new, extra second-to-last argument, nilslast:bit, which indicates where NIL values are to be sorted: at the beginning, or at the end. The old behavior is when for algebra.sort, the value of nilslast is equal to the value or the reverse:bit parameter, for algebra.firstn when the value of nilslast is opposite to the value of the asc:bit parameter. Build Environment * We now use the C11 atomic operations if available, although libatomic_ops is still used if present and not disabled on the configure command line. * We now also build the Python 3 integration packages for Fedora, Debian, Ubuntu, and Windows (not for Debian 8, Ubuntu 14.04, or EPEL 6 and 7 which are too old). SELinux Support * There is one tunable parameter, mserver5_can_read_home, which can be set using "setsebool -P mserver5_can_read_home=true" to allow an mserver5 process started by monetdbd under the control of systemd to read files in users' home directories. Merovingian * Added a new database property named `profilerbeatfreq`. Its value, <freq>, should be an integer and, if it is set the command `profilerstart` will call stethoscope with this value as the argument to -b. This will instruct stethoscope to collect heartbeat events from the MonetDB server every <freq> milliseconds. Please note that no type checking is done, and if <freq> is not an integer, stethoscope will silently ignore it. Client Package * The mclient program can now be quit by typing quit or exit on the command-line. This makes it easier for novice users to quit the program if they do not know that the standard command to quit is \q. Mapi Library * Removed function mapi_cache_shuffle. * Removed function mapi_stream_query. Stream Library * The functions mnstr_write{Sht,Int,Lng,Hge} and their Array variants now swap bytes of the written values if the stream is set to swap bytes (i.e. big-endian on a little-endian machine and v.v.). * Removed defines ST_ASCII, ST_BIN, ST_READ, ST_WRITE from stream.h. * Changed function mnstr_set_byteorder(stream, char) to mnstr_set_bigendian(stream, bool) where the second argument should be `true' for specifying that the stream is bigendian. This sets an internal flag whether or not to swap bytes which can be retrieved with mnstr_get_swapbytes(stream). * Changed function `int mnstr_byteorder(stream)' to `bool mnstr_get_swapbytes(stream)'; it now returns whether bytes are swapped. * Changed function `int mnstr_type(stream)' to `bool mnstr_isbinary(stream)'. MonetDB Common * During processing, we now try to maintain the smallest and largest values that occur in a BAT. This information is not saved across server restarts. * Added an extra argument of type `bool' to the atom to string and string to atom methods to indicate whether the string is for internal use (casting the value to or from a string) of external use (to be printed). * Implemented a nilslast option for BATfirstn. If set, NILs come last in the ordering that BATfirstn simulates, so non-NIL values are preferentially returned. The old behavior can be obtained by setting nilslast to !asc(ending). * Implemented a nilslast option for BATsort. This option should be equal to the reverse option for stable sort (it is not implemented for stable sort), but can be different from reverse for non-stable sort. The functions BATsort and GDKqsort have extra parameters, the function GDKqsort_rev has been removed (superseded by GDKqsort with the new `reverse' parameter). * The BUNtail, BUNtvar, BUNtloc, and BUNtpos macros (and Tloc and Tpos) now return a `void *' instead of a `char *'. * Function PROPdestroy now takes a BAT* parameter. * A whole bunch of functions now take a bool argument instead of an int argument. Some functions now return a bool instead of an int. In all these cases, the int was used as a Boolean value. SQL Frontend * Implemented two-argument least/greatest functions. If one argument is NULL, the functions return the other value, otherwise they return the least/greatest of the two arguments. * Implemented SQL2003 natural logarithm function: ln(num_expr). * Function octet_length now also accepts a BLOB argument and returns the length of the BLOB (i.e. the same as length when called with a BLOB argument). * Implemented length(blob) and comparison between blobs. * Implemented a two argument function log which calculates the logarithm of the first argument using the second argument as base. * Next to the functions log (natural logarithm) and log10 (base 10 logarithm), there is now also log2 (base 2 logarithm). * Removed unused global SQL variable "history". * Allow usage of custom expressions in GROUP BY and PARTITION BY clauses. The same expressions can be used in the projection phase as long as it is exactly the same used in the grouping clause (textual lookup is performed). As an example the query: SELECT col1*2 FROM t1 GROUP BY col1*2; is correct, while SELECT sin(col1+5) FROM t1 GROUP BY col1*2; is wrong. * Allow usage of WITH clauses in insert, update, delete and merge statements. * Implemented merge statements from SQL:2003 standard. Using a source relation R, a target table T is updating depending on the result of the merge. In case of a match, the table T's row is either updated or deleted with R's row. In a non-match case, R's row is inserted into T. The grammar is the follows: > MERGE INTO target [ [AS] ident ] USING source ON search_condition { WHEN MATCHED [ AND search_condition ] THEN { UPDATE SET assignment_list | DELETE } } | { WHEN NOT MATCHED [ AND search_condition ] THEN INSERT [ column_list ] [ { DEFAULT VALUES | VALUES row_values } ] } * Added possibility to change the schema of a table with the syntax: > ALTER TABLE [ IF EXISTS ] qname SET SCHEMA ident * Added optional alias option for target table in update and delete statements: (e.g. UPDATE target AS alias SET col1=1 WHERE alias.col2 < 5) * Improved and extended storagemodel functionality. Changes include: * Added views sys."tablestorage" and sys."schemastorage" for easy and quick assesment of aggregated storage usage per table or per schema. * Excluded listing system tables in views sys."storage", sys."tablestorage" and sys."schemastorage". You can still use sys."storage"() to query storage of system tables and columns. * In procedure sys.storagemodelinit() when populating table sys.storagemodelinput it now excludes system tables as system tables are not useful to be modeled for storagesize by application users. It now also computes and populates the atomwidth column more correctly for variable size data types (strings, blobs) when the table has representative data in those columns. * System function sys.columnsize() has been corrected for types: tinyint, real, date, time, timetz, sec_interval, month_interval, decimal, uuid, mbr, char, clob, json, url, blob, geometry and geometrya. For variable size data types (varchar, char, clob, json, url, blob, geometry, geometrya) it now returns the columnsize excluding the variable heapsize. The heapsize is retrievable via sys.heapsize(). * System function sys.heapsize() has been corrected for all variable size data types: varchar, char, clob, json, url, blob, geometry and geometrya. * System function sys.imprintsize() has been corrected. The imprintsize depends on the width (1 or 2 or 4 or 8 or 16 bytes) of the data type. Also instead of 12% it now uses 20%, which is a better estimate. * System function sys.storagemodel() has been removed as it outputs the same data as view sys.storagemodel. Use view sys.storagemodel instead. * Corrected views sys.storagemodel and sys.tablestoragemodel by returning a computed orderidxsize when the count has been changed in the sys.storagemodelinput table. Also the views now return data ordered by schema, table and column names. * Extended view sys.tablestoragemodel with column: "storages". Besides columns also keys (primary, foreign and unique keys) and indexes (ordered, imprints) use storage, so the "storages" count can be higher than the number of columns per table. * Corrected the data type of columns "schema", "table", "column", "type", "mode" and location in table sys.storagemodelinput and functions sys."storage"(), sys."storage"(sname), sys."storage"(sname, tname) and sys."storage"(sname, tname, cname) from string into varchar(1024). Consequently also the views based on the table or functions will be reporting varchar(1024) as column meta data instead of clob for those columns. This allows faster querying and reporting by generic SQL programs which treat clob querying different from varchar columns. * Extended windowing functions catalog with SQL standard semantics. Standard aggregation functions can now be used in windowing functions: (avg,max,min,sum,prod,count). Other windowing specific functions were also implemented: (percent_rank,cume_dist,ntile, lag,lead,first_value,last_value,nth_value). * The standard frame specification was implemented for aggregation functions as well as first_value, last_value and nth_value functions. The available frames are rows, range and groups. Bounds can be unbounded (partition limit), the current row, a fixed number of rows (constant), or variable (column as input). (e.g SELECT COUNT(col1) OVER (PARTITION BY col2 ORDER BY col3 RANGE BETWEEN UNBOUNDED PRECEDING AND CURRENT ROW) FROM t1;). * Added WINDOW keyword which is optionally provided after the FROM clause, with window specifications used in the projection (e.g SELECT SUM(col1) OVER w1, LAST_VALUE(col2) OVER w2 FROM t1 WINDOW w1 AS (ROWS BETWEEN 5 PRECEDING AND 0 FOLLOWING), w2 AS (w1);). * Our previous partitioning implementation didn’t impose order in the input. With this reexamination, partitioning now imposes ascending order by default, thus pairing with the industry standard implementation. * Implemented X'...' style binary string literals. * Implemented U&'...' Unicode character string literals and U&"..." Unicode delimited identifiers, including UESCAPE. For the string literals, you can have U&'...' '...' '...' UESCAPE '...' where the escape must be as single character and the other '...' strings are also Unicode character string literals. For now, these latter strings also undergo C-style backslash interpretation. * Implemented PostgreSQL-like E'...' strings. The strings can contain C-style backslash escapes. The old format strings '...' currently still also accept C-style escapes, but that feature will be removed in a future release. * Implemented the NULLS FIRST and NULLS LAST option to ORDER BY. The default is NULLS FIRST for ASC(ending) and NULLS LAST for DESC(ending). * Added possibility to rename SQL schemas, tables and columns with ALTER statements: > ALTER SCHEMA [ IF EXISTS ] ident RENAME TO ident > ALTER TABLE [ IF EXISTS ] qname RENAME TO ident > ALTER TABLE [ IF EXISTS ] qname RENAME [ COLUMN ] ident TO ident Also added optional IF EXISTS clause to other existing ALTER TABLE statements. * The sql SAMPLE syntax is extended to include an optional integral SEED parameter that sets the seed for the internal random number generator in the sample algorithm. When the seed is set, the user can obtain a reproducible sample set from a static dataset with the same sample query. The new syntax is SELECT ... FROM ... WHERE ... SAMPLE <expr> [SEED <integer>] * Implemented a version of COPY INTO that reads/writes files from/to the client instead of doing it in the server. This has the advantage that COPY INTO is then no longer restricted to only the "super user" monetdb, nor only to absolute file names. The syntax to have the server communicate with the client for file content is COPY INTO table FROM file ON CLIENT ...; and COPY query INTO file ON CLIENT ...;. This also works for COPY BINARY INTO. There is also the possibility to specify that files are to be read/written by the server by using ON SERVER. This is also the default when ON CLIENT or ON SERVER is not specified. * The sys.functions table now has an extra column called "system" of type BOOLEAN whose value is TRUE for system functions (i.e. functions that should not be dumped). The table sys.systemfunctions has been changed to a view and is now officially deprecated. * Extended merge tables with partitioning using a predicate: > CREATE MERGE TABLE [ IF NOT EXISTS ] table_name (... columns ...) [ PARTITION BY { RANGE | VALUES } { ON '(' column_name ')' | USING '(' expression ')' } ] The partitioning can occur by range or list of values using one of the table's columns or an expression. The domain of the partitioning scheme will be covered by each child table added with alter statements depending if the table is partitioned by range or list of values: > ALTER TABLE merge_table_name { ADD | SET } TABLE child_table_name AS PARTITION IN '(' expression [ ',' ... ] ')' [ WITH NULL ] > ALTER TABLE merge_table_name { ADD | SET } TABLE child_table_name AS PARTITION BETWEEN { RANGE MINVALUE | expression } AND { RANGE MAXVALUE | expression } [ WITH NULL ] > ALTER TABLE merge_table_name { ADD | SET } TABLE child_table_name AS PARTITION WITH NULL When updating a single partition range/list of values, the SET keyword should be used instead of ADD. INSERT, UPDATE and DELETE statements are possible on partitioned tables with corresponding validation on each partition domain. The system tables sys.table_partitions, sys.range_partitions and sys.value_partitions were added to store meta-information about each existing partition. Bug Fixes * 2403: stream: http read support in streams * 2416: GDK: file/dir creation mask * 2478: SQL: rename columns * 2496: SQL: implement greatest / least * 3384: Auxiliary window functions * 3416: log() SQL function only supports natural logarithm * 3448: Make lbatomic_ops optional if installed * 3520: Add support for scalar function LN(num) * 3530: sigabort on window aggr function * 3567: Add support for: ALTER TABLE [schema1.]oldtablename RENAME [TO] [schema2.]newtablename * 3743: DELETE FROM merge table not supported. * 3832: Cannot use expressions in GROUP BY clause * 3946: Expose C log2 function to SQL * 4056: mapi_timeout is not implemented * 6181: comments cause error with return in function * 6326: Eliminate unused UNION tables * 6347: Mserver returns with "memory exhausted in" error if query size exceeds 2.5MB * 6475: Remove unused SQL global variable "history" * 6588: Table aliasing are not supported in SQL update queries * 6591: Name mangling in the SQL catalog results in ambiguity * 6608: Sqlitelogictest error message in group by with coalesce * 6609: Rename Schema in MonetDB * 6636: sys.queue() extension: allow DBA to see all running queries * 6647: Add suport to Python 3 on Windows * 6669: COPY [xxx RECORDS] INTO foo FROM STDIN ... doesn't work without specifying nr of to be copied records * 6671: Error when running user function on merge table with remote part * 6674: R UDF with Date type fails to convert RAPI * 6676: Max data length of 2048 for column sys._tables.query is too small for the actual data (2811 for view sys.ids) * 6678: Binding NULL parameter to parametrized query results in syntax error on execution * 6680: Copy cannot open CSV file if systemctl enabled * 6684: Inserting into a replica table crashes the server * 6685: adding a view to a merge table gives unexpected behaviour * 6690: Unable to fetch empty string with ODBC driver * 6696: Re-use of MAL explain plans with complex type coercions

2 1

Re: Get rid of MCstartMaintenance and MCexitMaintenance.
by aris 19 Sep '19

19 Sep '19

On 19-09-19 12:00, developers-list-request(a)monetdb.org wrote: > Send developers-list mailing list submissions to > developers-list(a)monetdb.org > > To subscribe or unsubscribe via the World Wide Web, visit > https://www.monetdb.org/mailman/listinfo/developers-list > or, via email, send a message with subject or body 'help' to > developers-list-request(a)monetdb.org > > You can reach the person managing the list at > developers-list-owner(a)monetdb.org > > When replying, please edit your Subject line so it is more specific > than "Re: Contents of developers-list digest..." > > > Today's Topics: > > 1. Re: MonetDB: mosaic - Get rid of MCstartMaintenance and > MCexitMainte... (Martin Kersten) > > > ---------------------------------------------------------------------- > > Message: 1 > Date: Wed, 18 Sep 2019 17:13:33 +0200 > From: Martin Kersten <martin.kersten(a)cwi.nl> > To: developers-list(a)monetdb.org > Subject: Re: MonetDB: mosaic - Get rid of MCstartMaintenance and > MCexitMainte... > Message-ID: <90f5c40e-3c1c-5299-83cb-621660f3960c(a)cwi.nl> > Content-Type: text/plain; charset=utf-8; format=flowed > > I suggest to keep these functions available in the code base. > > They may be needed soon and are easier to control then > using monetdb stop <db> Sure but not in the mosaic feature branch. They are not needed there as I am now using proper GDK style bat level locks. But I guess it wouldn't hurt if somebody cherry picks the corresponding change sets directly into the default branch. But I am too busy currently :) > > > > On 17/09/2019 17:41, Aris Koning wrote: >> Changeset: b9c74a8bec29 for MonetDB >> URL: https://dev.monetdb.org/hg/MonetDB?cmd=changeset;node=b9c74a8bec29 >> Modified Files: >> clients/Tests/exports.stable.out >> monetdb5/mal/mal_client.c >> monetdb5/mal/mal_client.h >> Branch: mosaic >> Log Message: >> >> Get rid of MCstartMaintenance and MCexitMaintenance. >> >> >> diffs (99 lines): >> >> diff --git a/clients/Tests/exports.stable.out b/clients/Tests/exports.stable.out >> --- a/clients/Tests/exports.stable.out >> +++ b/clients/Tests/exports.stable.out >> @@ -1449,7 +1449,6 @@ str MBMuniform(bat *ret, oid *base, lng >> int MCactiveClients(void); >> str MCawakeClient(int id); >> void MCcloseClient(Client c); >> -str MCexitMaintenance(Client cntxt); >> Client MCforkClient(Client father); >> Client MCgetClient(int id); >> bool MCinit(void); >> @@ -1457,7 +1456,6 @@ Client MCinitClient(oid user, bstream *f >> int MCinitClientThread(Client c); >> int MCpushClientInput(Client c, bstream *new_input, int listing, char *prompt); >> int MCshutdowninprogress(void); >> -str MCstartMaintenance(Client cntxt, lng timeout, int abort); >> void MCstopClients(Client c); >> str MCsuspendClient(int id); >> int MCvalid(Client c); >> diff --git a/monetdb5/mal/mal_client.c b/monetdb5/mal/mal_client.c >> --- a/monetdb5/mal/mal_client.c >> +++ b/monetdb5/mal/mal_client.c >> @@ -518,64 +518,6 @@ MCawakeClient(int id) >> } >> >> /* >> - * Server maintenance modes without stopping it completery calls for care. >> - * Any new client connection should be temporily be refused >> - * and all clients should stop at a MAL instruction. >> - * In rare cases, all running queries should be aborted. >> - */ >> -str >> -MCstartMaintenance(Client cntxt, lng timeout, int abort) >> -{ >> - int active=1; >> - Client c = mal_clients; >> - (void) abort; >> - >> - MT_lock_set(&mal_contextLock); >> - for(c= mal_clients +1; c < mal_clients+MAL_MAXCLIENTS; c++) >> - if( cntxt != c) >> - c-> itrace = 'S'; >> - // wait for all running users to stop >> - while (active && timeout > 0){ >> - active = 0; >> - for(c= mal_clients +1; c < mal_clients+MAL_MAXCLIENTS; c++) >> - if( cntxt!= c && c->mode == RUNCLIENT){ >> - if (abort ){ >> - c->mode = FINISHCLIENT; >> - active++; >> - } else >> - if( cntxt != c) >> - active += c->itrace == 'S'; >> - } else >> - if (cntxt!= c && c->mode == FINISHCLIENT) >> - active++; >> - if( active == 0) >> - break; >> - MT_sleep_ms(1000); >> - timeout--; >> - } >> - if( timeout == 0 && active){ >> - for(c= mal_clients +1; c < mal_clients+MAL_MAXCLIENTS; c++) >> - if( c->itrace == 'W' || c->itrace == 'S') >> - c->itrace = 0; >> - throw(MAL,"clients.startmaintenance","timeout"); >> - } >> - return MAL_SUCCEED; >> -} >> - >> -str >> -MCexitMaintenance(Client cntxt) >> -{ >> - Client c = mal_clients; >> - (void) cntxt; >> - >> - for(c= mal_clients +1; c < mal_clients+MAL_MAXCLIENTS; c++) >> - if( c->itrace == 'W' || c->itrace == 'S') >> - c->itrace = 0; >> - MT_lock_unset(&mal_contextLock); >> - return MAL_SUCCEED; >> -} >> - >> -/* >> * Input to be processed is collected in a Client specific buffer. It >> * is filled by reading information from a stream, a terminal, or by >> * scheduling strings constructed internally. The latter involves >> diff --git a/monetdb5/mal/mal_client.h b/monetdb5/mal/mal_client.h >> --- a/monetdb5/mal/mal_client.h >> +++ b/monetdb5/mal/mal_client.h >> @@ -201,8 +201,6 @@ mal_export str MCsuspendClient(int i >> mal_export str MCawakeClient(int id); >> mal_export int MCpushClientInput(Client c, bstream *new_input, int listing, char *prompt); >> mal_export int MCvalid(Client c); >> -mal_export str MCstartMaintenance(Client cntxt, lng timeout, int abort); >> -mal_export str MCexitMaintenance(Client cntxt); >> >> mal_export str PROFinitClient(Client c); >> mal_export str PROFexitClient(Client c); >> _______________________________________________ >> checkin-list mailing list >> checkin-list(a)monetdb.org >> https://www.monetdb.org/mailman/listinfo/checkin-list >> > > ------------------------------ > > Subject: Digest Footer > > _______________________________________________ > developers-list mailing list > developers-list(a)monetdb.org > https://www.monetdb.org/mailman/listinfo/developers-list > > > ------------------------------ > > End of developers-list Digest, Vol 78, Issue 8 > **********************************************

1 0

Re: MonetDB: mosaic - Get rid of MCstartMaintenance and MCexitMainte...
by Martin Kersten 18 Sep '19

18 Sep '19

I suggest to keep these functions available in the code base. They may be needed soon and are easier to control then using monetdb stop <db> On 17/09/2019 17:41, Aris Koning wrote: > Changeset: b9c74a8bec29 for MonetDB > URL: https://dev.monetdb.org/hg/MonetDB?cmd=changeset;node=b9c74a8bec29 > Modified Files: > clients/Tests/exports.stable.out > monetdb5/mal/mal_client.c > monetdb5/mal/mal_client.h > Branch: mosaic > Log Message: > > Get rid of MCstartMaintenance and MCexitMaintenance. > > > diffs (99 lines): > > diff --git a/clients/Tests/exports.stable.out b/clients/Tests/exports.stable.out > --- a/clients/Tests/exports.stable.out > +++ b/clients/Tests/exports.stable.out > @@ -1449,7 +1449,6 @@ str MBMuniform(bat *ret, oid *base, lng > int MCactiveClients(void); > str MCawakeClient(int id); > void MCcloseClient(Client c); > -str MCexitMaintenance(Client cntxt); > Client MCforkClient(Client father); > Client MCgetClient(int id); > bool MCinit(void); > @@ -1457,7 +1456,6 @@ Client MCinitClient(oid user, bstream *f > int MCinitClientThread(Client c); > int MCpushClientInput(Client c, bstream *new_input, int listing, char *prompt); > int MCshutdowninprogress(void); > -str MCstartMaintenance(Client cntxt, lng timeout, int abort); > void MCstopClients(Client c); > str MCsuspendClient(int id); > int MCvalid(Client c); > diff --git a/monetdb5/mal/mal_client.c b/monetdb5/mal/mal_client.c > --- a/monetdb5/mal/mal_client.c > +++ b/monetdb5/mal/mal_client.c > @@ -518,64 +518,6 @@ MCawakeClient(int id) > } > > /* > - * Server maintenance modes without stopping it completery calls for care. > - * Any new client connection should be temporily be refused > - * and all clients should stop at a MAL instruction. > - * In rare cases, all running queries should be aborted. > - */ > -str > -MCstartMaintenance(Client cntxt, lng timeout, int abort) > -{ > - int active=1; > - Client c = mal_clients; > - (void) abort; > - > - MT_lock_set(&mal_contextLock); > - for(c= mal_clients +1; c < mal_clients+MAL_MAXCLIENTS; c++) > - if( cntxt != c) > - c-> itrace = 'S'; > - // wait for all running users to stop > - while (active && timeout > 0){ > - active = 0; > - for(c= mal_clients +1; c < mal_clients+MAL_MAXCLIENTS; c++) > - if( cntxt!= c && c->mode == RUNCLIENT){ > - if (abort ){ > - c->mode = FINISHCLIENT; > - active++; > - } else > - if( cntxt != c) > - active += c->itrace == 'S'; > - } else > - if (cntxt!= c && c->mode == FINISHCLIENT) > - active++; > - if( active == 0) > - break; > - MT_sleep_ms(1000); > - timeout--; > - } > - if( timeout == 0 && active){ > - for(c= mal_clients +1; c < mal_clients+MAL_MAXCLIENTS; c++) > - if( c->itrace == 'W' || c->itrace == 'S') > - c->itrace = 0; > - throw(MAL,"clients.startmaintenance","timeout"); > - } > - return MAL_SUCCEED; > -} > - > -str > -MCexitMaintenance(Client cntxt) > -{ > - Client c = mal_clients; > - (void) cntxt; > - > - for(c= mal_clients +1; c < mal_clients+MAL_MAXCLIENTS; c++) > - if( c->itrace == 'W' || c->itrace == 'S') > - c->itrace = 0; > - MT_lock_unset(&mal_contextLock); > - return MAL_SUCCEED; > -} > - > -/* > * Input to be processed is collected in a Client specific buffer. It > * is filled by reading information from a stream, a terminal, or by > * scheduling strings constructed internally. The latter involves > diff --git a/monetdb5/mal/mal_client.h b/monetdb5/mal/mal_client.h > --- a/monetdb5/mal/mal_client.h > +++ b/monetdb5/mal/mal_client.h > @@ -201,8 +201,6 @@ mal_export str MCsuspendClient(int i > mal_export str MCawakeClient(int id); > mal_export int MCpushClientInput(Client c, bstream *new_input, int listing, char *prompt); > mal_export int MCvalid(Client c); > -mal_export str MCstartMaintenance(Client cntxt, lng timeout, int abort); > -mal_export str MCexitMaintenance(Client cntxt); > > mal_export str PROFinitClient(Client c); > mal_export str PROFexitClient(Client c); > _______________________________________________ > checkin-list mailing list > checkin-list(a)monetdb.org > https://www.monetdb.org/mailman/listinfo/checkin-list >

1 0

Re: ALTER TABLE ALTER COLUMN SET STORAGE
by aris 14 Sep '19

14 Sep '19

Hi Dan, On 12-09-19 11:01, developers-list-request(a)monetdb.org wrote: > Send developers-list mailing list submissions to > developers-list(a)monetdb.org > > To subscribe or unsubscribe via the World Wide Web, visit > https://www.monetdb.org/mailman/listinfo/developers-list > or, via email, send a message with subject or body 'help' to > developers-list-request(a)monetdb.org > > You can reach the person managing the list at > developers-list-owner(a)monetdb.org > > When replying, please edit your Subject line so it is more specific > than "Re: Contents of developers-list digest..." > > > Today's Topics: > > 1. Re: ALTER TABLE ALTER COLUMN SET STORAGE (Daniel Zvinca) > 2. Re: ALTER TABLE ALTER COLUMN SET STORAGE (Daniel Zvinca) > > > ---------------------------------------------------------------------- > > Message: 1 > Date: Wed, 11 Sep 2019 13:53:27 +0300 > From: Daniel Zvinca <daniel.zvinca(a)logbis.com> > To: "Communication channel for developers of the MonetDB suite." > <developers-list(a)monetdb.org> > Subject: Re: ALTER TABLE ALTER COLUMN SET STORAGE > Message-ID: > <CALxHsPEdkfOUKjy-xV5GDYHFry=SAst5qzZM5rk8JvFHqk1FNQ(a)mail.gmail.com> > Content-Type: text/plain; charset="utf-8" > > Thank you so much for your answer, Aris. > > The good news is that compression is considered and is going to be part of > MonetDB in two releases or so (one year). First release is probably much closer to half a year. > > I will try to see if I can use the current code in a custom build, I am > quite curious how that will affect performance. > To be honest I don't expect performance issues, nowadays even on SSD era, > it is still faster to read compressed data and decompress in memory if the > right ratio is there, of course. And I do expect decent compression ratio > on most of the data. The idea is that between every memory fetch you have quite some CPU cycles at your disposal which can be used to perform either (potentially) more expensive algebraic operations directly on compressed data or to just decompress data. This is the general principle that implies that at least lightweight compression can be used to accelerate queries. > > However, at this stage the MOSAIC's dual compressed - uncompressed > storage will obviously not give me the gain I need. Yet, it is interesting > to understand at least how query performance might look in the future. Again this feature is still under discussion. But I don't think it is unlikely that it will enter into Mosaic. > > Bad news is of course the compression feature is going to happen in ... one > year. But if it comes also with support for compressing in memory results > (I know, it wasnt promised), it might worth to wait. We're working it ☺. More remarks below on your other email... > > Best regards, > > > > > > > On Wed, Sep 11, 2019 at 12:18 PM aris <aris.koning(a)monetdbsolutions.com> > wrote: > >> >> >> -------- Forwarded Message -------- >> Subject: Re: ALTER TABLE ALTER COLUMN SET STORAGE >> Date: Tue, 10 Sep 2019 15:15:09 +0200 >> From: aris <aris.koning(a)monetdbsolutions.com> >> <aris.koning(a)monetdbsolutions.com> >> To: developers-list-request(a)monetdb.org >> >> Hi Daniel, >> >> On 10-09-19 12:00, developers-list-request(a)monetdb.org wrote: >> >> Send developers-list mailing list submissions to >> developers-list(a)monetdb.org >> >> To subscribe or unsubscribe via the World Wide Web, visit >> https://www.monetdb.org/mailman/listinfo/developers-list >> or, via email, send a message with subject or body 'help' to >> developers-list-request(a)monetdb.org >> >> You can reach the person managing the list at >> developers-list-owner(a)monetdb.org >> >> When replying, please edit your Subject line so it is more specific >> than "Re: Contents of developers-list digest..." >> >> >> Today's Topics: >> >> 1. ALTER TABLE ALTER COLUMN SET STORAGE (Daniel Zvinca) >> >> >> ---------------------------------------------------------------------- >> >> Message: 1 >> Date: Tue, 10 Sep 2019 12:40:22 +0300 >> From: Daniel Zvinca <daniel.zvinca(a)logbis.com> <daniel.zvinca(a)logbis.com> >> To: developers-list(a)monetdb.org >> Subject: ALTER TABLE ALTER COLUMN SET STORAGE >> Message-ID: >> <CALxHsPHEEExozCgdp_srxQm2tFtFOqGOGNqBTw2UhTBwWZu93g(a)mail.gmail.com> >> <CALxHsPHEEExozCgdp_srxQm2tFtFOqGOGNqBTw2UhTBwWZu93g(a)mail.gmail.com> >> Content-Type: text/plain; charset="utf-8" >> >> Hello, >> >> I am interested to find out more about ALTER TABLE ALTER COLUMN SET STORAGE >> feature and how is that related to compression. >> >> As far as I understood this is related to an active development branch, >> called MOSAIC which was never merged with any of the previous MonetDB >> versions. Obviously, compression is an important feature columnar databases >> are providing for data storage and manipulation. A module like MOSAIC that >> seems to allow several compression techniques, would be an interesting >> option. >> >> Yes, compression a.k.a. Mosaic is a going to be a new feature in MonetDB. >> Although the feature won't be included in the upcoming November release. >> Most likely, you can expect the feature in the first release after the >> November release. But Mosaic is a somewhat big undertaking. our current >> road map is probably covering multiple future MonetDB releases before all >> envisioned compression features are available in MonetDB. The first >> milestone in the current road map is to apply a single compression >> technique on an entire column. But one of the next milestone is to >> partition a column into variable-sized compression blocks. Within each >> block a particular compression is applied. >> >> >> First question I have: Can MOSAIC extension be used (sources added and >> custom compiled) with success for any of its proposed codecs with any of >> the newest versions (Apr2019 +). I mean without affecting any of embedded, >> capi, rapi and pyapi modules, which all exchange data with external >> libraries. >> >> If by this you mean you want to import the mosaic module as an external >> library into an existing release out of the box, then the answer is no. >> There are some slight modifications in the GDK layer to accommodate the >> Mosaic module. And to interact with it from SQL, there are also some code >> changes in the SQL layer. But besides those dependencies, I don't expect >> any issue with the particular (x)api frameworks. But nothing is guaranteed >> obviously. It sounds like you want to hack-back port it into custom builds >> of earlier releases. I wouldn't give it a zero change of success but I do >> wish you much luck :) >> >> >> A quick read of MOSAIC code made me understand that this compression can be >> applied only on readonly PERSISTENT columns. That means that I would loose >> the major benefit of compression that I mostly need during importing stage. >> Sure I can imagine a controlled batching import scheme that would append >> data to tables and when it reaches certain threshold table is made >> readonly, then compressed, then added to a merged table, but this looks >> quite of a scenario. Am I wrong, can MOSAIC be used in a different >> scenario? >> >> Your observation about the joint life cycle of a Mosaic structure and its >> original column file is correct: currently Mosaic adds a compressed >> representation next to the existing uncompressed column. For the first >> milestone on the Mosaic road map we want to successfully apply compression >> on READ-ONLY pre-existing columns where the purpose of compression is to >> potentially accelerate analytical queries on these columns. However we are >> still looking into potentially freeing the uncompressed column once a >> compressed Mosaic heap is available. This would accommodate compression for >> the more traditional sake of limiting memory- and/or disk footprint. >> >> >> I can understand reasons behind compressing only PERSISTENT bats, yet I am >> wondering if TRANSIENT bats can also benefit from it especially for 1. >> result building stage (server-client or embedded version) or 2. for remote >> connections when data is transferred for merging operations. >> >> Regarding to above question, are there any chances that you would consider >> keeping compressed results in memory? Sure I can use instead disk >> temporary tables for subsequent manipulation, but for performance reasons >> in memory compressed results would be way faster. Actually, when embedded >> version provides a result set, it stays valid till the user releases it, >> why not to be able to also use that for possible subsequent SQL operations >> that do not fit into a CTE scenario. That would provide superior >> flexibility and memory management to CTE mechanism. Temporary results can >> be developed in steps, they can be accessed directly at any time as >> convenient as temporary views in CTE, but without the burden of possible >> temporary bats that are not released till one CTE ends. >> >> I think it is an interesting idea. But I think it is part of a more >> general goal/problem of how to handle updates on compressed data. There are >> internal discussions on this topic. But whatever the outcome, this will be >> only relevant for a much later milestone on the road map. >> >> >> >> Thank you, >> Dan >> >> >> Hope it helps. >> >> Kind regards, >> >> Aris >> >> >>

2 1

Re: ALTER TABLE ALTER COLUMN SET STORAGE
by aris 12 Sep '19

12 Sep '19

-------- Forwarded Message -------- Subject: Re: ALTER TABLE ALTER COLUMN SET STORAGE Date: Tue, 10 Sep 2019 15:15:09 +0200 From: aris <aris.koning(a)monetdbsolutions.com> To: developers-list-request(a)monetdb.org Hi Daniel, On 10-09-19 12:00, developers-list-request(a)monetdb.org wrote: > Send developers-list mailing list submissions to > developers-list(a)monetdb.org > > To subscribe or unsubscribe via the World Wide Web, visit > https://www.monetdb.org/mailman/listinfo/developers-list > or, via email, send a message with subject or body 'help' to > developers-list-request(a)monetdb.org > > You can reach the person managing the list at > developers-list-owner(a)monetdb.org > > When replying, please edit your Subject line so it is more specific > than "Re: Contents of developers-list digest..." > > > Today's Topics: > > 1. ALTER TABLE ALTER COLUMN SET STORAGE (Daniel Zvinca) > > > ---------------------------------------------------------------------- > > Message: 1 > Date: Tue, 10 Sep 2019 12:40:22 +0300 > From: Daniel Zvinca <daniel.zvinca(a)logbis.com> > To: developers-list(a)monetdb.org > Subject: ALTER TABLE ALTER COLUMN SET STORAGE > Message-ID: > <CALxHsPHEEExozCgdp_srxQm2tFtFOqGOGNqBTw2UhTBwWZu93g(a)mail.gmail.com> > Content-Type: text/plain; charset="utf-8" > > Hello, > > I am interested to find out more about ALTER TABLE ALTER COLUMN SET > STORAGE > feature and how is that related to compression. > > As far as I understood this is related to an active development branch, > called MOSAIC which was never merged with any of the previous MonetDB > versions. Obviously, compression is an important feature columnar > databases > are providing for data storage and manipulation. A module like MOSAIC that > seems to allow several compression techniques, would be an interesting > option. Yes, compression a.k.a. Mosaic is a going to be a new feature in MonetDB. Although the feature won't be included in the upcoming November release. Most likely, you can expect the feature in the first release after the November release. But Mosaic is a somewhat big undertaking. our current road map is probably covering multiple future MonetDB releases before all envisioned compression features are available in MonetDB. The first milestone in the current road map is to apply a single compression technique on an entire column. But one of the next milestone is to partition a column into variable-sized compression blocks. Within each block a particular compression is applied. > > First question I have: Can MOSAIC extension be used (sources added and > custom compiled) with success for any of its proposed codecs with any of > the newest versions (Apr2019 +). I mean without affecting any of embedded, > capi, rapi and pyapi modules, which all exchange data with external > libraries. If by this you mean you want to import the mosaic module as an external library into an existing release out of the box, then the answer is no. There are some slight modifications in the GDK layer to accommodate the Mosaic module. And to interact with it from SQL, there are also some code changes in the SQL layer. But besides those dependencies, I don't expect any issue with the particular (x)api frameworks. But nothing is guaranteed obviously. It sounds like you want to hack-back port it into custom builds of earlier releases. I wouldn't give it a zero change of success but I do wish you much luck :) > > A quick read of MOSAIC code made me understand that this compression > can be > applied only on readonly PERSISTENT columns. That means that I would loose > the major benefit of compression that I mostly need during importing > stage. > Sure I can imagine a controlled batching import scheme that would append > data to tables and when it reaches certain threshold table is made > readonly, then compressed, then added to a merged table, but this looks > quite of a scenario. Am I wrong, can MOSAIC be used in a different > scenario? Your observation about the joint life cycle of a Mosaic structure and its original column file is correct: currently Mosaic adds a compressed representation next to the existing uncompressed column. For the first milestone on the Mosaic road map we want to successfully apply compression on READ-ONLY pre-existing columns where the purpose of compression is to potentially accelerate analytical queries on these columns. However we are still looking into potentially freeing the uncompressed column once a compressed Mosaic heap is available. This would accommodate compression for the more traditional sake of limiting memory- and/or disk footprint. > > I can understand reasons behind compressing only PERSISTENT bats, yet I am > wondering if TRANSIENT bats can also benefit from it especially for 1. > result building stage (server-client or embedded version) or 2. for remote > connections when data is transferred for merging operations. > > Regarding to above question, are there any chances that you would consider > keeping compressed results in memory? Sure I can use instead disk > temporary tables for subsequent manipulation, but for performance reasons > in memory compressed results would be way faster. Actually, when embedded > version provides a result set, it stays valid till the user releases it, > why not to be able to also use that for possible subsequent SQL operations > that do not fit into a CTE scenario. That would provide superior > flexibility and memory management to CTE mechanism. Temporary results can > be developed in steps, they can be accessed directly at any time as > convenient as temporary views in CTE, but without the burden of possible > temporary bats that are not released till one CTE ends. I think it is an interesting idea. But I think it is part of a more general goal/problem of how to handle updates on compressed data. There are internal discussions on this topic. But whatever the outcome, this will be only relevant for a much later milestone on the road map. > > > Thank you, > Dan Hope it helps. Kind regards, Aris >

2 2

ALTER TABLE ALTER COLUMN SET STORAGE
by Daniel Zvinca 10 Sep '19

10 Sep '19

Hello, I am interested to find out more about ALTER TABLE ALTER COLUMN SET STORAGE feature and how is that related to compression. As far as I understood this is related to an active development branch, called MOSAIC which was never merged with any of the previous MonetDB versions. Obviously, compression is an important feature columnar databases are providing for data storage and manipulation. A module like MOSAIC that seems to allow several compression techniques, would be an interesting option. First question I have: Can MOSAIC extension be used (sources added and custom compiled) with success for any of its proposed codecs with any of the newest versions (Apr2019 +). I mean without affecting any of embedded, capi, rapi and pyapi modules, which all exchange data with external libraries. A quick read of MOSAIC code made me understand that this compression can be applied only on readonly PERSISTENT columns. That means that I would loose the major benefit of compression that I mostly need during importing stage. Sure I can imagine a controlled batching import scheme that would append data to tables and when it reaches certain threshold table is made readonly, then compressed, then added to a merged table, but this looks quite of a scenario. Am I wrong, can MOSAIC be used in a different scenario? I can understand reasons behind compressing only PERSISTENT bats, yet I am wondering if TRANSIENT bats can also benefit from it especially for 1. result building stage (server-client or embedded version) or 2. for remote connections when data is transferred for merging operations. Regarding to above question, are there any chances that you would consider keeping compressed results in memory? Sure I can use instead disk temporary tables for subsequent manipulation, but for performance reasons in memory compressed results would be way faster. Actually, when embedded version provides a result set, it stays valid till the user releases it, why not to be able to also use that for possible subsequent SQL operations that do not fit into a CTE scenario. That would provide superior flexibility and memory management to CTE mechanism. Temporary results can be developed in steps, they can be accessed directly at any time as convenient as temporary views in CTE, but without the burden of possible temporary bats that are not released till one CTE ends. Thank you, Dan

1 0

New Nov2019 stable branch created
by Sjoerd Mullender 05 Sep '19

05 Sep '19

I have created a new stable branch for the next feature release. This branch was created in order to stabilize the code before the feature release. From now on *only* bug fixes may go to this new stable branch. I reserve the right to undo any changes that do not comply with this rule. I also reserve the right to admit changes that are not strictly bug fixes. The name for the branch is Nov2019. So in each of your stable checkout directories you can do hg pull hg update -rNov2019 to update to this new branch. -- Sjoerd Mullender

1 0

Announcement: New Apr2019-SP1 Bugfix release of MonetDB suite
by Sjoerd Mullender 02 Sep '19

02 Sep '19

The MonetDB team at CWI/MonetDB BV is pleased to announce the Apr2019-SP1 bugfix release of the MonetDB suite of programs. More information about MonetDB can be found on our website at <https://www.monetdb.org/>. For details on this release, please see the release notes at <https://www.monetdb.org/Downloads/ReleaseNotes>. As usual, the download location is <https://dev.monetdb.org/downloads/>. Apr 2019-SP1 bugfix release (11.33.11) Build Environment * Removed restriction on using combinations of --enable-assert, --enable-debug, and --enable-optimize. --enable-debug adds a -g (or -even -g3) option, --enable-debug=gdb adds a -ggdb3 flag (for GCC); --enable-optimize adds a bunch of optimization flags; --disable-debug (or --enable-debug=no) removes any -g flags; --disable-optimize removes any -O flags; --enable-optimize=auto leaves the optimization flags untouched. MonetDB Common * We now look at the limits imposed by cgroups and the setrlimit system call to initialize some internal values related to how much (virtual) memory we think is available. SQL Frontend * Many fixes for using the server in a highly concurrent setting with many clients doing read-only queries mixed with clients doing updating queries. * Changed the internal representation of RANGE MINVALUE and RANGE MAXVALUE in MERGE TABLE partitions. Before, the limits were represented by the smallest and largest value of the domain of the column, now these are represented by a NULL value. This has the added benefit that if you use TO RANGE MAXVALUE, the largest value of the domain is included in the partition (before it wasn't). Also, these new limits now work for any ordered type, so including VARCHAR. Bug Fixes * 6576: Sqlitelogictest aritmetic expressions with negative numbers handling * 6697: Duplicate expressions not eliminated with long CASE statement * 6701: When changing the schema name of a table, referencing rows from sys.columns, sys.keys and more tables are not removed * 6703: SQL optimizer enters loop and goes into stack overflow * 6706: prepare doesn't recognize merge statement * 6712: Where clause with cast ignores sub-select * 6713: COPY INTO FROM 'file.xz' does not work * 6714: Assertion failure in rel_select.c for correlated subquery in aggregation query (with group by and having and order by and limit) * 6715: Assertion failure in rel_bin.c for MERGE INTO command * 6716: COPY INTO does not load UTF8 encoded text * 6718: Can't copy table into another table with constraints (mkey.bulk_rotate_xor_hash) * 6720: Compilation issues on gcc 9.1.1 [-Werror=stringop-truncation] * 6721: Possibly incorrect call to pcre_exec * 6725: Prepare statement on non-existing function crashes * 6728: SELECT DISTINCT not removing duplicates * 6729: Creating a table with duplicate column name should not be possible * 6730: sys.stop or sys.pause a INSERT query triggers "BATproject: does not match always" * 6736: Unexpected query result with merge tables and view after upgrade * 6738: issue with window functions and hugeint type coercion * 6741: report no such table when execute “alter table tb1 rename to tb2” according to using mapi * 6744: CREATE SEQUENCE seq; fails * 6748: MonetDB "forgets" columns after ALTER TABLE statements * 6749: mserver5 restart aborts/segfaults after dropping column with constraint * 6751: ODBC driver(11.33.3) Seg Faults when "fn ucase" is used in SQL

1 0

Alter question about MonetDB
by 角 06 Aug '19

06 Aug '19

Hi,All: I am a MonetDB user from China. I have some troubles when I was using MonetDB. I use mclient and enter this sql : "alter table t1 rename to t2；", while this mclient is alive,I use anthor mclient to connect the same database as the former and I enter this sql : "select * from t2" then I get an error : no such table. So how can I use alter command to make all mclients take effect?

1 0

Re: developers-list Digest, Vol 76, Issue 1
by Odin Loch 20 Jun '19

20 Jun '19

Where?! Need more details.. ________________________________ From: developers-list <developers-list-bounces+odin.loch=outlook.com.au(a)monetdb.org> on behalf of developers-list-request(a)monetdb.org <developers-list-request(a)monetdb.org> Sent: Thursday, June 20, 2019 8:00:02 PM To: developers-list(a)monetdb.org Subject: developers-list Digest, Vol 76, Issue 1 Send developers-list mailing list submissions to developers-list(a)monetdb.org To subscribe or unsubscribe via the World Wide Web, visit https://www.monetdb.org/mailman/listinfo/developers-list or, via email, send a message with subject or body 'help' to developers-list-request(a)monetdb.org You can reach the person managing the list at developers-list-owner(a)monetdb.org When replying, please edit your Subject line so it is more specific than "Re: Contents of developers-list digest..." Today's Topics: 1. UTF-8 data: copy into fails (Arjen P. de Vries) 2. Re: UTF-8 data: copy into fails (Panagiotis Koutsourakis) 3. Re: UTF-8 data: copy into fails (Arjen P. de Vries) ---------------------------------------------------------------------- Message: 1 Date: Wed, 19 Jun 2019 23:06:48 +0200 From: "Arjen P. de Vries" <arjen(a)acm.org> To: developers-list(a)monetdb.org Subject: UTF-8 data: copy into fails Message-ID: <CAM_um4c2Qyab7Y9E9efQqdgfGPBmSr7bns=49t+-goGCqiyNZw(a)mail.gmail.com> Content-Type: text/plain; charset="utf-8" I filed a bug report but did not see any response so maybe better mention it here: https://www.monetdb.org/bugzilla/show_bug.cgi?id=6716 Definitely would be nice to know if you can reproduce the bug as reported. Groetjes, Arjen _____ Please excuse brevity - sent from a mobile.

1 0

UTF-8 data: copy into fails
by Arjen P. de Vries 20 Jun '19

20 Jun '19

I filed a bug report but did not see any response so maybe better mention it here: https://www.monetdb.org/bugzilla/show_bug.cgi?id=6716 Definitely would be nice to know if you can reproduce the bug as reported. Groetjes, Arjen _____ Please excuse brevity - sent from a mobile.

3 2

Re: MonetDB: default - rewrite pairs of x >(=) lb and x <(=) hb into...
by Martin Kersten 27 Apr '19

27 Apr '19

:) On 27/04/2019 14:53, Niels Nes wrote: > Changeset: b9a037bfe7db for MonetDB > URL: https://dev.monetdb.org/hg/MonetDB?cmd=changeset;node=b9a037bfe7db > Added Files: > sql/test/BugTracker-2019/Tests/duplicates-not-eliminated-long-CASE-stmt.Bug-6697.stable.out > Modified Files: > sql/common/sql_types.c > sql/server/rel_optimizer.c > sql/test/Tests/systemfunctions.stable.out > sql/test/Tests/systemfunctions.stable.out.int128 > Branch: default > Log Message: > > rewrite pairs of x >(=) lb and x <(=) hb into between calls

1 0

"ALTER TABLE name SET SCHEMA new_schema" support
by Anton Kravchenko 17 Apr '19

17 Apr '19

Is a table moving between schemas going to be supported in the next feature release? Thanks, Anton

2 1

Possible bug in pcre.c
by Roberto Cornacchia 08 Apr '19

08 Apr '19

Hi, I was looking at this piece of code (end of pcrejoin loop), and I have the feeling something isn't right. Inside the body of if (nl == 0) a condition over lastl is checked. However, it seems to me that lastl gets a useful value only when nl > 0. If (nl == 0) then lastl must be 0. <loop> { /* actual body */ .... /* track result properties */ if (BATcount(r1) > 0) { if (lastl + 1 != lo) r1->tseqbase = oid_nil; if (nl == 0) { r2->trevsorted = 0; if (lastl > lo) { r1->tsorted = 0; r1->tkey = 0; } else if (lastl < lo) { r1->trevsorted = 0; } else { r1->tkey = 0; } } } APPEND(r1, lo); APPEND(r2, ro); lastl = lo; nl++; }

2 2

Compiler optimizations and MonetDB performance
by aris 28 Mar '19

28 Mar '19

Hi all, One of our customers has pointed out that GCC 7+ has a reasonable large number of additions to their optimization algorithms and this customer was wondering if the MonetDB R&D community is aware of these compiler improvements and what their combined effect is on the performance of MonetDB. This customer seems to be contemplating to adopt the build process of MonetDB into their own build & deployment pipelines. So they are interested in the most optimal build signature for MonetDB on their Centos environment. Somebody already pointed out that given the kernel architecture of MonetDB, there is a chance that for instance, (new) inner loop optimizations can be useful for MonetDB since we put quite some effort into deciding which specialized algorithm to use for a particular relational algebraic operation before we dispatch the workload to the designated tight and specialized inner loop that does the actual work. Is anybody aware of existing evaluations/benchmarks related to different compiler, compiler settings and architectures/instruction set extensions? Or is it planned? Or should it be planned :) Kind regards, Aris

2 1

Re: MonetDB: subquery - return single column on in queries (with one...
by Martin Kersten 17 Mar '19

17 Mar '19

making progres ;) > On 17 Mar 2019, at 11:58, Niels Nes <commits+niels=cwi.nl(a)monetdb.org> wrote: > > Changeset: 4ca5e0d20906 for MonetDB > URL: https://dev.monetdb.org/hg/MonetDB?cmd=changeset;node=4ca5e0d20906 > Modified Files: > sql/server/rel_select.c > sql/server/rel_unnest.c > sql/test/BugTracker-2014/Tests/in_incorrect_multi.Bug-3462.stable.err > Branch: subquery > Log Message: > > return single column on in queries (with one column on the left) > > > diffs (127 lines): > > diff --git a/sql/server/rel_select.c b/sql/server/rel_select.c > --- a/sql/server/rel_select.c > +++ b/sql/server/rel_select.c > @@ -1832,7 +1832,7 @@ rel_compare(sql_query *query, sql_rel *r > } else { > /* first try without current relation, too see if there > are correlations with the outer relation */ > - sql_rel *r = rel_subquery(query, NULL, ro, ek, 0);//APPLY_JOIN); > + sql_rel *r = rel_subquery(query, NULL, ro, ek, 0); > > /* NOT handled filter case */ > if (ro2) > @@ -1842,7 +1842,7 @@ rel_compare(sql_query *query, sql_rel *r > sql->session->status = 0; > sql->errstr[0] = 0; > query_push_outer(query, rel); > - r = rel_subquery(query, NULL, ro, ek, 0);// is_sql_sel(f)?APPLY_LOJ:APPLY_JOIN); > + r = rel_subquery(query, NULL, ro, ek, 0); > query_pop_outer(query); > > /* get inner queries result value, ie > @@ -2137,7 +2137,7 @@ rel_logical_value_exp(sql_query *query, > } else { > /* first try without current relation, too see if there > are correlations with the outer relation */ > - sql_rel *r = rel_subquery(query, NULL, ro, ek, 0);//APPLY_JOIN); > + sql_rel *r = rel_subquery(query, NULL, ro, ek, 0); > sql_exp *rs2 = NULL; > > /* correlation, ie return new relation */ > @@ -2147,8 +2147,10 @@ rel_logical_value_exp(sql_query *query, > /* reset error */ > sql->session->status = 0; > sql->errstr[0] = 0; > - if (!r) > + if (!r) { > + assert(0); > r = rel_subquery(query, *rel, ro, ek, is_sql_sel(f)?APPLY_LOJ:APPLY_JOIN); > + } > > /* get inner queries result value, ie > get last expression of r */ > @@ -2477,7 +2479,7 @@ rel_logical_value_exp(sql_query *query, > if ((!orel || (is_project(orel->op) && !is_processed(orel) && !orel->l && list_empty(orel->exps))) && !query_has_outer(query)) > orel = *rel = rel_project_exp(sql->sa, exp_atom_bool(sql->sa, 1)); > > - ek.card = card_set; > + ek.card = card_column; > if (is_sql_sel(f) && orel && is_project(orel->op) && !is_processed(orel)) { > needproj = 1; > pexps = orel->exps; > @@ -2811,13 +2813,13 @@ rel_logical_exp(sql_query *query, sql_re > if (!rel) > return NULL; > } > + return rel; > /* > rel = rel_logical_exp(query, rel, lo, f); > if (!rel) > return NULL; > return rel_logical_exp(query, rel, ro, f); > */ > - return rel; > } > case SQL_FILTER: > /* [ x,..] filter [ y,..] */ > @@ -2922,6 +2924,8 @@ rel_logical_exp(sql_query *query, sql_re > } > > } > + if (list_length(ll) > 1) > + ek.card = card_relation; > > /* list of values or subqueries */ > if (n->type == type_list) { > @@ -3021,7 +3025,7 @@ rel_logical_exp(sql_query *query, sql_re > } > if (l_used) > rel_join_add_exp(sql->sa, left, e); > - else if (!is_select(left->op)) > + else if (!is_select(left->op) || rel_is_ref(left)) > left = rel_select(sql->sa, left, e); > else > rel_select_add_exp(sql->sa, left, e); > @@ -3100,7 +3104,7 @@ rel_logical_exp(sql_query *query, sql_re > list *pexps = NULL; > int needproj = 0, exists=(sc->token == SQL_EXISTS); > > - ek.card = card_set; > + ek.card = card_column; > if (orel && is_project(orel->op) && !is_processed(orel)) { > needproj = 1; > pexps = orel->exps; > @@ -5777,7 +5781,7 @@ rel_value_exp2(sql_query *query, sql_rel > if (r) { > sql_exp *e; > > - if (ek.card <= card_column && is_project(r->op) && list_length(r->exps) > 1) > + if (ek.card <= card_set && is_project(r->op) && list_length(r->exps) > 1) > return sql_error(sql, 02, SQLSTATE(42000) "SELECT: subquery must return only one column"); > e = _rel_lastexp(sql, r); > > diff --git a/sql/server/rel_unnest.c b/sql/server/rel_unnest.c > --- a/sql/server/rel_unnest.c > +++ b/sql/server/rel_unnest.c > @@ -679,6 +679,9 @@ rel_unnest_dependent(mvc *sql, sql_rel * > return rel_unnest_dependent(sql, rel); > } > > + if (r && is_base(r->op)) /* table functions need dependent implementation */ > + return rel; > + > /* fallback */ > if ((ad = rel_dependent_var(sql, rel->l, rel->r)) != NULL) > rel = rel_general_unnest(sql, rel, ad); > diff --git a/sql/test/BugTracker-2014/Tests/in_incorrect_multi.Bug-3462.stable.err b/sql/test/BugTracker-2014/Tests/in_incorrect_multi.Bug-3462.stable.err > --- a/sql/test/BugTracker-2014/Tests/in_incorrect_multi.Bug-3462.stable.err > +++ b/sql/test/BugTracker-2014/Tests/in_incorrect_multi.Bug-3462.stable.err > @@ -31,7 +31,7 @@ stderr of test 'in_incorrect_multi.Bug-3 > MAPI = (monetdb) /var/tmp/mtest-27483/.s.monetdb.35395 > QUERY = SELECT * FROM SYS.ARGS > WHERE FUNC_ID NOT IN (SELECT * FROM SYS.FUNCTIONS); > -ERROR = !IN: inner query should return a single column > +ERROR = !SELECT: subquery must return only one column > CODE = 42000 > > > _______________________________________________ > checkin-list mailing list > checkin-list(a)monetdb.org > https://www.monetdb.org/mailman/listinfo/checkin-list

2 1

New Apr2019 stable branch created
by Sjoerd Mullender 18 Jan '19

18 Jan '19

I have created a new stable branch for the next feature release. This branch was created in order to stabilize the code before the feature release. From now on *only* bug fixes may go to this new stable branch. I reserve the right to undo any changes that do not comply with this rule. I also reserve the right to admit changes that are not strictly bug fixes. The name for the branch is Apr2019. So in each of your stable checkout directories you can do hg pull hg update -rApr2019 to update to this new branch. -- Sjoerd Mullender

1 0

Announcement: New Aug2018-SP2 Bugfix release of MonetDB suite
by Sjoerd Mullender 14 Jan '19

14 Jan '19

The MonetDB team at CWI/MonetDB BV is pleased to announce the Aug2018-SP2 bugfix release of the MonetDB suite of programs. More information about MonetDB can be found on our website at <https://www.monetdb.org/>. For details on this release, please see the release notes at <https://www.monetdb.org/Downloads/ReleaseNotes>. As usual, the download location is <https://dev.monetdb.org/downloads/>. Aug 2018-SP2 bugfix release (11.31.13) SQL Frontend * Disabled function sys.getcontent(url). Bug Fixes * 6643: schema name qualifier in create global temporary table json.table6643 ... is not honered * 6645: optimizer treats all the functions with no or constant parameters as constant * 6650: PREPARE SQL statement fails to compile user defined functions with parameter/s * 6651: Multi-column IN clause for subquery produces wrong results * 6653: CREATE TABLE accepts empty table/column name * 6654: Incorrect handling of 'TRUE' in compound select * 6657: Restart sequence with a non atomic sub-query cardinality gives wrong error message * 6660: GRANTing a ROLE is not idempotent * 6662: Concurrency Conflicts Exception string "!transaction is aborted because of concurrency conflicts, will ROLLBACK instead" not shown on prompt. * 6664: mserver5 crash: infinite recursive happens at rel_bin.c:489 * 6665: Creation of serial types does not accept negative numbers * 6666: COPY INTO from .. LOCKED doubles input data * 6668: The SAMPLE key word doesn't work in a subquery. * 6669: COPY [xxx RECORDS] INTO foo FROM STDIN ... doesn't work without specifying nr of to be copied records * 6672: SQLGetData with SQL_C_WCHAR string truncation and invalid StrLen_or_Ind value

1 0

How to perform copy into in multiple clients at the same time?
by Yinjie Lin 02 Jan '19

02 Jan '19

Hi, I'm new to MonetDB and I think this is a great project! I have a problem the same as this one: https://stackoverflow.com/questions/9179265/how-would-monetdb-perform-with-… , namely copy data into tables from multiple clients at the same time. As the stackoverflow question suggests, MonetDB uses Optimistic Concurrency Control, but the behaviour is different from my expectation. For example, say I have a table `t` without any constraints, and I run the following queries: client1 client2 ------------------------------------------------------------------- start transaction; insert into t values (1); start transaction; insert into t values (2); commit; commit; I would expect both commits succeed since the rows to be inserted do not conflict with each other and do not change original rows in the table. And this is the behaviour of some databases with OCC, e.g. TiDB (a MySQL compatible database). But MonetDB would rollback the second transaction. So my question is: is it possible to change this behaviour by modifying the code? If so, could you give me some guidance or suggestion? Thanks in advance! Best Regards, Yinjie Lin

4 5

Announcement: New Aug2018-SP1 Bugfix release of MonetDB suite
by Sjoerd Mullender 17 Oct '18

17 Oct '18

The MonetDB team at CWI/MonetDB BV is pleased to announce the Aug2018-SP1 bugfix release of the MonetDB suite of programs. More information about MonetDB can be found on our website at <https://www.monetdb.org/>. For details on this release, please see the release notes at <https://www.monetdb.org/Downloads/ReleaseNotes>. As usual, the download location is <https://dev.monetdb.org/downloads/>. Aug 2018-SP1 bugfix release (11.31.11) Build Environment * On Ubuntu 18.10 (Cosmic Cuttlefish), the libmonetdb5-server-bam package cannot be built because of an incompatibility in the libbam library (it cannot be used in a shared object. Whole System * Some subtle dependencies between RPMs have been fixed. Bug Fixes * 6640: timestamp_to_str returning incorrectly adjusted results * 6641: race condition in SQL UDF with update * 6642: Hanging query * 6648: key property potentially wrong after type conversion * 6649: Projection inside within transaction gives wrong results

1 0

mvc_create_table new parameter
by Panagiotis Koutsourakis 03 Sep '18

03 Sep '18

Hi all, trying to merge the data vaults branch with the default branch I noticed we have a new parameter named 'properties' to the 'mvc_create_table' function. Is there somewhere a comment or even better some documentation about what it means and what the allowed values are? Thanks, Panos.

1 1

Announcement: New Aug2018 Feature release of MonetDB suite
by Sjoerd Mullender 31 Aug '18

31 Aug '18

The MonetDB team at CWI/MonetDB BV is pleased to announce the Aug2018 feature release of the MonetDB suite of programs. More information about MonetDB can be found on our website at <https://www.monetdb.org/>. For details on this release, please see the release notes at <https://www.monetdb.org/Downloads/ReleaseNotes>. As usual, the download location is <https://dev.monetdb.org/downloads/>. Aug 2018 feature release (11.31.7) MonetDB5 Server * The lsst module was moved to a separate repository (https://dev.monetdb.org/hg/MonetDB-lsst/) Build Environment * Build the MonetDB-cfitsio RPM and libmonetdb5-server-cfitsio Debian/Ubuntu package. * On Windows, the separate MonetDB5-Geom installer has been incorporated into the main MonetDB5-SQL installer and is therefore no longer available as a separate download. Merovingian * Added a "logrotate" configuration file. See /etc/logrotate.d/monetdbd. * Changed the monetdb profilerstart command to be more robust. If the server or stethoscope crashed before, the pid file is still there, so the next time we try to start stethoscope, it will fail. Now the profilerstart command will check if a stethoscope process with the recorded pid is running. If not, we start stethoscope, assuming that something went wrong before. * Changed the monetdb stop command to try to stop stethoscope before stoping the server. The error conditions that can arrise from attempting to stop stethoscope are: + The database is not running. + The profilerlogpath is not set. + The profiler.pid file does not exist or is inaccessible. + The contents of the profiler.pid are not valid. + Shutdown of stethoscope did not succeed. + Removing the profiler.pid file failed. In all the cases, the attempt to stop the server can continue normally, so we actually ignore any errors that rise from the attempt to stop stethoscope. Client Package * ODBC: Implemented SQL_ATTR_QUERY_TIMEOUT parameter in SQLSetStmtAttr. * ODBC SQLGetInfo now returns a positive numeric value for InfoTypes: SQL_MAX_COLUMN_NAME_LEN, SQL_MAX_DRIVER_CONNECTIONS, SQL_MAX_IDENTIFIER_LEN, SQL_MAX_PROCEDURE_NAME_LEN, SQL_MAX_SCHEMA_NAME_LEN, SQL_MAX_TABLE_NAME_LEN and SQL_MAX_USER_NAME_LEN. * Added a '-f rowcount' option in mclient to repress printing the actual data of a resultset, but only print the number of returned tuples Stream Library * Added support for lz4 compressed files in the stream library MonetDB Common * Hash indexes are now persistent across server restarts. * The macros bunfastapp and tfastins and variants no longer set the dirty flag of the heap they write to. This now needs to be done separately (and preferably outside of the inner loop). * Removed batDirty flag from BAT record. Its function is completely superseded by batDirtydesc and the dirty flags on the various heaps. * Removed "masksize" argument of function BAThash. * A whole bunch of functions that took an int argument that was used as a Boolean (true/false) value now take a value of type bool. The functions BATkeyed, BATordered and BATordered_rev now return a bool instead of an int. * Removed the tdense property: it's function is completely replaced by whether or not tseqbase is equal to oid_nil. Testing Environment * Removed helper programs Mtimeout and MkillUsers: they have long been superseded by timeout handling by Mtest.py itself. SQL Frontend * Removed deprecated table producing system functions: sys.dependencies_columns_on_functions() sys.dependencies_columns_on_indexes() sys.dependencies_columns_on_keys() sys.dependencies_columns_on_triggers() sys.dependencies_columns_on_views() sys.dependencies_functions_on_functions() sys.dependencies_functions_on_triggers() sys.dependencies_keys_on_foreignkeys() sys.dependencies_owners_on_schemas() sys.dependencies_schemas_on_users() sys.dependencies_tables_on_foreignkeys() sys.dependencies_tables_on_functions() sys.dependencies_tables_on_indexes() sys.dependencies_tables_on_triggers() sys.dependencies_tables_on_views() sys.dependencies_views_on_functions() sys.dependencies_views_on_triggers() They are replaced by new system dependency_* views: sys.dependency_args_on_types sys.dependency_columns_on_functions sys.dependency_columns_on_indexes sys.dependency_columns_on_keys sys.dependency_columns_on_procedures sys.dependency_columns_on_triggers sys.dependency_columns_on_types sys.dependency_columns_on_views sys.dependency_functions_on_functions sys.dependency_functions_on_procedures sys.dependency_functions_on_triggers sys.dependency_functions_on_types sys.dependency_functions_on_views sys.dependency_keys_on_foreignkeys sys.dependency_owners_on_schemas sys.dependency_schemas_on_users sys.dependency_tables_on_foreignkeys sys.dependency_tables_on_functions sys.dependency_tables_on_indexes sys.dependency_tables_on_procedures sys.dependency_tables_on_triggers sys.dependency_tables_on_views sys.dependency_views_on_functions sys.dependency_views_on_procedures sys.dependency_views_on_views * Implemented group_concat(X,Y) aggregate function which also concatenates a column of strings X, but using a supplied string Y as the separator. This function is also a SQL extension. * Implemented group_concat(X) aggregate function which concatenates a column of strings using a comma as a separator. This function is not featured in the SQL standard. Bug Fixes * 4020: Importing timestamp with zone from copy into * 6506: Improper performance counters * 6556: Sqlitelogictest division by zero on COALESCE call * 6564: Changes to the Remote Table definition * 6575: Sqlitelogictest crash on groupby query with coalesce call * 6579: Sqlitelogic test infinite loop while compiling SQL query * 6586: Sqlitelogictest crash on complex aggregation query * 6593: Poor performance with like operator and escape clause * 6596: Multicolumn aggregation very slow after ANALYZE when persistent hashes are enabled * 6605: Sqlitelogictest set queries with wrong results * 6606: Misleading parameter name in generate_series function * 6610: Sqlitelogictest algebra.rangejoin undefined * 6611: Cannot compile with GCC 8.1 and --enable-debug=no * 6612: Implement BLOB handling in python UDFs * 6618: dependency column on sequence violated by DROP SEQUENCE * 6621: SELECT FROM REMOTE TABLE WHERE <> returns wrong results * 6624: "Cannot use non GROUP BY column in query results without an aggregate function" when using aggregate function in both HAVING and ORDER BY clauses. * 6625: OR in subselect causes the server to crash with segmentation fault * 6627: stddev_pop inconsistent behaviour * 6628: User cannot insert into own local temporary table * 6629: CREATE TABLE IF NOT EXISTS returns 42000! * 6630: Sqlitelogictest cast NULL to integer failing * 6632: Dataflow causes crash when THRnew fails * 6633: ILIKE clauses don't work on certain characters * 6635: monetdbd exits due to "Too many open files" error * 6637: Within a transaction, \d after an error causes mclient to exit * 6638: (sequences of) mkey.bulk_rotate_xor_hash() can generate NIL from non-NIL making multi-col joins return wrong results * 6639: COMMENT ON TABLE abc IS NULL invalidly sets the remark column to null where remark column is defined as NOT NULLable

1 0

New Aug2018 stable branch created
by Sjoerd Mullender 30 Jul '18

30 Jul '18

I have created a new stable branch for the next feature release. This branch was created in order to stabilize the code before the feature release. From now on *only* bug fixes may go to this new stable branch. I reserve the right to undo any changes that do not comply with this rule. I also reserve the right to admit changes that are not strictly bug fixes. The name for the branch is Aug2018. So in each of your stable checkout directories you can do hg pull hg update -rAug2018 to update to this new branch. -- Sjoerd Mullender

1 0

Python UDFs with blob column
by Florestan De Moor 19 Jul '18

19 Jul '18

Hello, I've been recently working with Python UDFs in MonetDB and I've noticed that returning a blob column from an array of python objects is not allowed. A look at the code confirmed that this feature is not implemented yet: "FIXME: check for byte array/or pickle object to string". I have thus written an implementation of this feature. Could you please tell me how I can submit my code for you to review it and consider to apply it? Thanks. Best regards, Florestan

3 3

Insert/update value manipulation
by Schediwy, Thomas (itsc) 11 Jul '18

11 Jul '18

Hi there, I am looking for the function where I can manipulate the values from an insert or update statement. For now, I am using a Solaris 10 system. But I havent found the functions where the insert or update will get done. I have looked trogh the files stream.c, mapi.c and mclient.c. But everytime I tried to change the values in the variables nothing happend. Can somebody please help me? Thanks in advise Thomas

3 5

Announcement: New Mar2018-SP1 Bugfix release of MonetDB suite
by Panagiotis Koutsourakis 20 Jun '18

20 Jun '18

The MonetDB team at CWI/MonetDB BV is pleased to announce the Mar2018-SP1 bugfix release of the MonetDB suite of programs. More information about MonetDB can be found on our website at <https://www.monetdb.org/>. For details on this release, please see the release notes at <https://www.monetdb.org/Downloads/ReleaseNotes>. As usual, the download location is <https://dev.monetdb.org/downloads/>. Mar 2018-SP1 bugfix release (11.29.7) Build Environment * On Linux and Unix, try not to link to libraries multiple times: that only causes the code (and worse, the variables in those libraries) to be included multiple times. On Windows, we do need to link to libraries multiple times (once for each DLL we create if the DLL references anything from the library) but that doesn't cause the linked library to be loaded multiple times. SQL Frontend * Corrected the definition of view: sys.ids. Bug Fixes * 6562: Sqlitelogictest crash on group by query with not in operator * 6565: Sqlitelogictest crash on complex select query with coalesce call * 6566: Sqlitelogictest unavailable calc.- MAL operations * 6568: Sqlitelogictest crash on complex case query * 6569: Sqlitelogictest select query with not between cause with wrong results * 6570: Sqlitelogictest select coalesce undefined calc * 6572: ordered index Error in optimizer garbageCollector * 6573: Sqlitelogictest crash on complex select query with case statement * 6574: server crashed could not find ordered index while creating a table * 6576: Sqlitelogictest aritmetic expressions with negative numbers handling * 6577: creating temp table kills performance of the original query * 6578: One two-tuple insert gives different results than two single inserts * 6581: Join condition errors. * 6583: Fixed size string concatenation with integer results in fixed size string of size 0 * 6584: SELECT FROM REMOTE TABLE WHERE IS NOT NULL produces wrong result * 6585: Nested Merge tables cause an infinite loop in rel_optimizer * 6587: Sqlitelogictest crash on complex case statement * 6589: Sqlitelogictest crash on complex on complex expression * 6594: Sqlitelogictest crash on complex case statement * 6595: Remote decimal division triggers assertion / returns wrong answer * 6598: Python 3.4 not supported (due to usage of Py_DecodeLocale) * 6600: Sqlitelogictest queries fail to execute * 6601: "where is null" clause on remote table causes problem with next query * 6602: Sqlitelogictest wrong results in IN query * 6603: Sqlitelogictest: Aggregation query with distinct clause produces duplicated rows

1 0

Vakantie
by Martin Kersten 22 Apr '18

22 Apr '18

wanneer was je op vakantie? Sent from my iPad > On 22 Apr 2018, at 11:54, Niels Nes <commits+niels=cwi.nl(a)monetdb.org> wrote: > > Changeset: 354f2d85e58b for MonetDB > URL: https://dev.monetdb.org/hg/MonetDB?cmd=changeset;node=354f2d85e58b > Added Files: > sql/test/BugTracker-2018/Tests/sqlitelogictest-complex-expression.Bug-6589.stable.err > sql/test/BugTracker-2018/Tests/sqlitelogictest-complex-expression.Bug-6589.stable.out > Modified Files: > sql/server/rel_select.c > Branch: Mar2018 > Log Message: > > fixes for bug 6589 > > > diffs (99 lines): > > diff --git a/sql/server/rel_select.c b/sql/server/rel_select.c > --- a/sql/server/rel_select.c > +++ b/sql/server/rel_select.c > @@ -2098,7 +2098,10 @@ rel_logical_value_exp(mvc *sql, sql_rel > > r = rel_value_exp(sql, &z, sval, f, ek); > if (l && r && IS_ANY(st->type->eclass)){ > - l = rel_check_type(sql, exp_subtype(r), l, type_equal); > + sql_exp *nl = rel_check_type(sql, exp_subtype(r), l, type_equal); > + if (nl != l && is_new) > + left = rel_project_exp(sql->sa, nl); > + l = nl; > if (l) > st = exp_subtype(l); > } > diff --git a/sql/test/BugTracker-2018/Tests/sqlitelogictest-complex-expression.Bug-6589.stable.err b/sql/test/BugTracker-2018/Tests/sqlitelogictest-complex-expression.Bug-6589.stable.err > new file mode 100644 > --- /dev/null > +++ b/sql/test/BugTracker-2018/Tests/sqlitelogictest-complex-expression.Bug-6589.stable.err > @@ -0,0 +1,35 @@ > +stderr of test 'sqlitelogictest-complex-expression.Bug-6589` in directory 'sql/test/BugTracker-2018` itself: > + > + > +# 11:47:28 > > +# 11:47:28 > "mserver5" "--debug=10" "--set" "gdk_nr_threads=0" "--set" "mapi_open=true" "--set" "mapi_port=31863" "--set" "mapi_usock=/var/tmp/mtest-18504/.s.monetdb.31863" "--set" "monet_prompt=" "--forcemito" "--dbpath=/home/niels/scratch/rc-old/Linux-x86_64/var/MonetDB/mTests_sql_test_BugTracker-2018" "--set" "embedded_c=true" > +# 11:47:28 > > + > +# builtin opt gdk_dbpath = /home/niels/scratch/rc-old/Linux-x86_64/var/monetdb5/dbfarm/demo > +# builtin opt gdk_debug = 0 > +# builtin opt gdk_vmtrim = no > +# builtin opt monet_prompt = > > +# builtin opt monet_daemon = no > +# builtin opt mapi_port = 50000 > +# builtin opt mapi_open = false > +# builtin opt mapi_autosense = false > +# builtin opt sql_optimizer = default_pipe > +# builtin opt sql_debug = 0 > +# cmdline opt gdk_nr_threads = 0 > +# cmdline opt mapi_open = true > +# cmdline opt mapi_port = 31863 > +# cmdline opt mapi_usock = /var/tmp/mtest-18504/.s.monetdb.31863 > +# cmdline opt monet_prompt = > +# cmdline opt gdk_dbpath = /home/niels/scratch/rc-old/Linux-x86_64/var/MonetDB/mTests_sql_test_BugTracker-2018 > +# cmdline opt embedded_c = true > +# cmdline opt gdk_debug = 553648138 > + > +# 11:47:28 > > +# 11:47:28 > "mclient" "-lsql" "-ftest" "-tnone" "-Eutf-8" "-i" "-e" "--host=/var/tmp/mtest-18504" "--port=31863" > +# 11:47:28 > > + > + > +# 11:47:28 > > +# 11:47:28 > "Done." > +# 11:47:28 > > + > diff --git a/sql/test/BugTracker-2018/Tests/sqlitelogictest-complex-expression.Bug-6589.stable.out b/sql/test/BugTracker-2018/Tests/sqlitelogictest-complex-expression.Bug-6589.stable.out > new file mode 100644 > --- /dev/null > +++ b/sql/test/BugTracker-2018/Tests/sqlitelogictest-complex-expression.Bug-6589.stable.out > @@ -0,0 +1,39 @@ > +stdout of test 'sqlitelogictest-complex-expression.Bug-6589` in directory 'sql/test/BugTracker-2018` itself: > + > + > +# 11:47:28 > > +# 11:47:28 > "mserver5" "--debug=10" "--set" "gdk_nr_threads=0" "--set" "mapi_open=true" "--set" "mapi_port=31863" "--set" "mapi_usock=/var/tmp/mtest-18504/.s.monetdb.31863" "--set" "monet_prompt=" "--forcemito" "--dbpath=/home/niels/scratch/rc-old/Linux-x86_64/var/MonetDB/mTests_sql_test_BugTracker-2018" "--set" "embedded_c=true" > +# 11:47:28 > > + > +# MonetDB 5 server v11.29.2 > +# This is an unreleased version > +# Serving database 'mTests_sql_test_BugTracker-2018', using 8 threads > +# Compiled for x86_64-unknown-linux-gnu/64bit > +# Found 15.387 GiB available main-memory. > +# Copyright (c) 1993 - July 2008 CWI. > +# Copyright (c) August 2008 - 2018 MonetDB B.V., all rights reserved > +# Visit https://www.monetdb.org/ for further information > +# Listening for connection requests on mapi:monetdb://xps13:31863/ > +# Listening for UNIX domain connection requests on mapi:monetdb:///var/tmp/mtest-18504/.s.monetdb.31863 > +# MonetDB/GIS module loaded > +# MonetDB/SQL module loaded > + > +Ready. > + > +# 11:47:28 > > +# 11:47:28 > "mclient" "-lsql" "-ftest" "-tnone" "-Eutf-8" "-i" "-e" "--host=/var/tmp/mtest-18504" "--port=31863" > +# 11:47:28 > > + > +#SELECT DISTINCT + - 82 * + 26 - 12 * + - 27 * + + CASE WHEN NOT NULL > +#IN ( - 69, ( + SUM ( DISTINCT CAST ( NULL AS INTEGER ) ) ) + + MAX ( - 6 ) + + - COUNT ( * ) / - - ( + 77 ) + 37, + 44, > +#- - 0 * + + 23, - - 86, 3, 12 ) THEN + 41 WHEN NOT - 21 / - 95 IS NOT NULL THEN 63 END AS col0; > +% .L42 # table_name > +% col0 # name > +% int # type > +% 1 # length > +[ NULL ] > + > +# 11:47:28 > > +# 11:47:28 > "Done." > +# 11:47:28 > > + > _______________________________________________ > checkin-list mailing list > checkin-list(a)monetdb.org > https://www.monetdb.org/mailman/listinfo/checkin-list

1 1

INSERT using multiple cores (inspired by COPY from CSV in LOCKED mode)
by Anton Kravchenko 12 Apr '18

12 Apr '18

Hi there, 1) It looks like INSERTing into tables is bounded by a single CPU. And it kills performance of a query that normally uses multiple CPUs... CREATE TEMP TABLE t1 AS (some 'select query' that uses multiple cores when run standalone) ON COMMIT PRESERVE ROWS; or INSERT into t1 (some 'select query' that uses multiple cores when run standalone); 2) Do you think you could implement INSERT with LOCKED option? (like it was done for COPY from CSV in LOCKED mode) Please let me know whether you think it's doable. Thank you, Anton

1 1

ordered index Error in optimizer garbageCollector
by Anton Kravchenko 06 Apr '18

06 Apr '18

Here is a reproducible example. create table t1(v0 int, v1 char(1)); insert into t1 values (1,'a'),(2,'b'),(3,'c'); create ordered index index_t1_v1 on t1(v1); create table t2(v1 char(1)); insert into t2 values ('a'); 1) gives "Error in optimizer garbageCollector" create temp table t3 as (select t1.v0 from t1,t2 where trim(t2.v1)=t1.v1) on commit preserve rows; 2) works as expected create temp table t3 as (select t1.v0 from t1,t2 where t2.v1=t1.v1) on commit preserve rows; 3) works as expected select t1.v0 from t1,t2 where trim(t2.v1)=t1.v1; 4) works as expected drop index index_t1_v1; create temp table t3 as (select t1.v0 from t1,t2 where trim(t2.v1)=t1.v1) on commit preserve rows; I am using the latest MonetDB version: MonetDB 5 server v11.29.3 "Mar2018" (64-bit, 128-bit integers) Copyright (c) 1993 - July 2008 CWI Copyright (c) August 2008 - 2018 MonetDB B.V., all rights reserved Visit https://www.monetdb.org/ for further information Found 31.3GiB available memory, 4 available cpu cores Libraries: libpcre: 8.32 2012-11-30 (compiled with 8.32) openssl: OpenSSL 1.0.2k 26 Jan 2017 (compiled with ) libxml2: 2.9.1 (compiled with 2.9.1) Compiled by: mockbuild@ (x86_64-redhat-linux-gnu) Compilation: gcc -std=gnu99 -O2 -g -pipe -Wall -Wp,-D_FORTIFY_SOURCE=2 -fexceptions -fstack-protector-strong --param=ssp-buffer-size=4 -grecord-gcc-switches -m64 -mtune=generic Linking : /usr/bin/ld -m elf_x86_64 -Wl,-z,relro -Wl,-Bsymbolic-functions Thanks, Anton

1 0

Re: tail heap hash found to be invalid on repeated DB loads
by Eyal Rozenberg 29 Mar '18

29 Mar '18

On 03/29/2018 03:55 PM, Sjoerd Mullender wrote: > On 29/03/18 11:02, Eyal Rozenberg wrote: > Indeed. Maybe at the first restart the hash will be overwritten, but > then probably not at any subsequent restart. But we still have to check > because we don't know (cannot know!) how the server exited last time. Yes, of course. But what I saw is that the check fails repeatedly. However, I'm having trouble reproducing this with 11.29.3 and on another machine, so I'll just let it go for now.

1 0

Announcement: New Mar2018 Feature release of MonetDB suite
by Sjoerd Mullender 29 Mar '18

29 Mar '18

The MonetDB team at CWI/MonetDB BV is pleased to announce the Mar2018 feature release of the MonetDB suite of programs. More information about MonetDB can be found on our website at <https://www.monetdb.org/>. For details on this release, please see the release notes at <https://www.monetdb.org/Downloads/ReleaseNotes>. As usual, the download location is <https://dev.monetdb.org/downloads/>. Some notable changes: - Code base requires a C99 compiler. - The internal representation of REAL and DOUBLE NULL values has changed. Before they were represented as the smallest C float/double finite value (-FLT_MAX and -DBL_MAX). Now they are represented by a NaN (not-a-number) value. This requires a database upgrade which is done automatically at startup. Be advised that it is recommended to make a backup before upgrading. - Various extensions to SQL have been implemented. For more details, see below. Mar 2018 feature release (11.29.3) MonetDB5 Server * Implemented function pcre.replace_first which is like pcre.replace, except it only replaces the first match. * The EXPLAIN command now shows all the MAL type resolutions, because in general users may not be aware of their signatures. It also simplifies programs to analyze such plans. * Implemented versions of group.(sub)group(done) that don't return a histogram. * Removed MAL functions streams.socketRead, streams.socketReadBytes, streams.socketWrite, and streams.socketWriteBytes. * Removed MAL functions streams.openRead(s:streams):streams and streams.openWrite(s:streams):streams. * Lots of changes to streamline the internal error handling. In principle, all errors should now include a SQLSTATE error code (see the SQL standard). Build Environment * Added the .pdb files needed for debug symbols to the Windows installer for MonetDB/SQL. . * mclient's execution time profiling options and output format have been changed. Instead of implicitly via the "--interactive"/"-i" option (or when using an interactive mclient console), execution time profiling is now controlled via an explicit "--timer=timermode"/"-t timermode" command-line option, or a "\t timermode" command in the interactive mclient console. The default (also in the interactive mclient console) is now timermode "none", i.e., no timing information is given. Timermode "clock" activates client-side wall-clock timing ("clk") in "human-friendly" format much like the interactive mode did before. Timermode "performance" also provides detailed server-side timings: "sql" is the time to parse the SQL query, optimize the logical relational plan and create the initial physical (MAL) plan; "opt" is the time to optimize the physical (MAL) plan; "run" is the time to execute the physical (MAL) plan. With timermode "performance" all server-side timings and the client-side wall-clock time are given in milliseconds (ms). Note that the client-measured wall-clock time "clk" is reported per query only when options "--interactive" or "--echo" are used, because only then does mclient send individual lines (statements) of the SQL script to the server. Otherwise, mclient sends the SQL script in large(r) batch(es) to the server, and, thus, only the total wall-clock time per batch is measured and reported. The server-measured detailed performance timings "sql", "opt", "run" are always measured and reported per query. Also, all timing information is now given on a separate line and sent to stderr rather than stdout. * Some types and constants were moved from configure (and hence monetdb_config.h) to gdk.h. In particular, the types "lng" and "ulng" have been moved and can therefore no longer be used by code that doesn't (ultimately) include gdk.h. Just use int64_t instead. A bunch of format defines have been removed: SZFMT, SSZFMT, PTRFMT, PDFMT. Just use the C standard codes for those (%zu, %zd, %p, %td). The define for printing a lng (LLFMT) was also moved. Use PRId64 for printing int64_t in code not using gdk.h. Removed all references to __int64 and long long (use int64_t instead). Merovingian * Add daemon commands for starting/stopping collection of profiler (stethoscope) logs. Client Package * ODBC: The driver function SQLProcedureColumns was implemented. * ODBC: Changed table types as used by SQLTables from "LOCAL TEMPORARY" and "GLOBAL TEMPORARY" to "LOCAL TEMPORARY TABLE" and "GLOBAL TEMPORARY TABLE" respectively. * ODBC: Initial support for the HUGEINT type in SQL was added. Any value with type HUGEINT can be retrieved in a C variable with type SQL_C_CHAR or SQL_C_WCHAR. A value of type HUGEINT can be retrieved in other C types as long as they fit, the largest C type supported being a 64 bit integer (equivalent to BIGINT). * The functions in the mapi library that require 64 bit integers now use the standard type int64_t instead of the non-standard mapi_int64. This requires a compilation environment that has the stdint.h include file (standardized in C99). Compilation of the library also requires the inttypes.h include file (also standardized in C99). * Add a new pretty printing option to stethoscope Running stethoscope with the flag -j will produce not pretty printed output (one json object per line). Running with the -y flag will produce pretty printed output. Running with neither will produce the legacy, line oriented format mapilib * The three mapi_explain* functions and mapi_trace don't return any useful information, so they now return void. * The functions mapi_error_str and mapi_result_error now return const char * instead of plain char * to indicate that the returned data belongs to the library and should not be changed or freed by the application. * New function const char *mapi_result_errorcode(MapiHdl) which returns the SQLSTATE code if available when an error has occurred. stream * The interface of mnstr_fgetpos and mnstr_fsetpos was changed to look more like the standard C functions fsetpos and fgetpos: they both have a second argument "pointer to fpos_t". * Removed function wbstream. * Removed functions udp_rastream and udp_wastream. * Removed functions socket_rstream and socket_wstream. * Removed functions append_wstream and append_wastream. * Removed functions mnstr_rstream and mnstr_wstream. utils * The function mcrypt_getHashAlgorithms now returns a static, constant string, so the result should not be modified or freed. MonetDB Common * Changed return type of function void_replace_bat from BUN to gdk_return: it now only returns whether the operation succeeded or not. * Changed the return type of BATroles from void to gdk_return: it can fail due to malloc failure. * Removed functions ALIGNsetH, ALIGNsetT, and CREATEview_ (mind the underscore). The first can easily be replace by using BAThseqbase (that's all it did), the second was only used once, and the third can be replace by VIEWcreate. * Removed unused functions BATmemsize and BATvmsize. * Removed the tnodense property: it was maintained but never actually used, not even stored. * The NIL representation of the internal flt and dbl types was changed from the smallest representable finite value to NaN (not-a-number). * Changed the interface of ATOMformat and VALformat: they now return a pointer to the allocated string. * The length "method" for atoms now returns a size_t, the "len" field of a ValRecord is now a size_t, the "size" field of the atomDesc structure is now unsigned short. * Removed the "align" field from the ATOM descriptor (atomDesc) structure. * The atomtostr and atomfromstr "methods" for atoms now return ssize_t and require a pointer to size_t for the size of the buffer. * The atom tostr and fromstr "methods" now always return -1 on error. A return value greater than 0 is normal, a return value of 0 is not normal, but technically not an error. Testing Environment * Added a --data_path option to Mtest.py that defines an external data repository. See the commit message of c484932c7fd8 for more info. SQL * Extended support to use CREATE ORDERED INDEX on columns of type: char, varchar, clob, blob, url, json, inet and uuid. * Added new system view: sys.ids which contains all database objects ids which can be used in sys.dependencies table. * Added new system view: sys.dependencies_vw which shows all data of sys.dependencies including names on objects, object types and dependency types. * Added 25 new system views for finding out dependencies between database objects. These new dependency views improve, extend and replace the 17 sys.dependencies_X_on_Y() functions as previously defined in 21_dependency_functions.sql. Those sys.dependencies_X_on_Y() functions are now marked as deprecated. * Added new system view: sys.roles which contains all defined roles. * Added new system view: sys.var_values which shows the values for system variables. * Added support for COMMENT ON statements using SQL syntax: COMMENT ON { SCHEMA | TABLE | VIEW | COLUMN | INDEX | SEQUENCE | FUNCTION | PROCEDURE | AGGREGATE | FILTER FUNCTION | LOADER } qname IS { 'my description text' | NULL | '' } ; For COLUMN the qname can be "table_name"."column_name" or fully qualified as in: "schema_name"."table_name"."column_name". For FUNCTION, PROCEDURE, AGGREGATE, FILTER FUNCTION and LOADER the qname may need to include the signature (argument types) to be able to differentiate between multiple overloaded functions which have the same name and schema. By specifying IS NULL or IS '' you remove the comment for the database object. If a database object is dropped, the associated comment is also removed. Note: it is not allowed or possible to add comments for temporary tables or objects in schema "tmp". The sql catalog has been extended with system table: sys.comments. The keyword 'COMMENT' has now become a reserved keyword. * Removed system function sys.environment(). It was a duplicate of system function sys.env(). * Implemented behavior for DROP SCHEMA my_schema RESTRICT command. Previously the RESTRICT keyword was accepted but not obeyed. It would always do a CASCADE operation. Also the default behavior of DROP SCHEMA my_schema command is now changed into RESTRICT behavior (was CASCADE). * The internal NULL representation of the REAL (FLOAT) and DOUBLE types was changed from the smallest representable finite value to NaN (not-a-number). * A column default value can be used in a UPDATE statement: UPDATE tname SET cname = DEFAULT, and INSERT statements: INSERT INTO tname VALUES (..., DEFAULT, ...) * Added support for TRUNCATE statements conforming to the SQL:2008 standard: TRUNCATE [ TABLE ] qname [ CONTINUE IDENTITY | RESTART IDENTITY ] [ RESTRICT | CASCADE ] In a TRUNCATE statement a 'CONTINUE IDENTITY' or 'RESTART IDENTITY' clause can be passed to restart or not, being the former the default one. The 'CASCADE' option instructs to truncate referencing table(s) also if the referencing table(s) have foreign key references to this table. The default behavior is 'RESTRICT'. Note: it is possible to use TRUNCATE statements in a transaction and thus to rollback the effects of a truncate. The keywords 'TRUNCATE' and 'CONTINUE' have now become reserved keywords. * Added possibility to GRANT or REVOKE a TRUNCATE privilege to a user or role. * Added possibility to define a TRIGGER on a TRUNCATE event. * Added possibility to specify 'OR REPLACE' in following CREATE commands: CREATE [ OR REPLACE ] VIEW qname ... CREATE [ OR REPLACE ] TRIGGER qname ... * Added possibility to specify 'IF EXIST' in following DROP commands: DROP AGGREGATE [ IF EXISTS ] qname ... DROP FUNCTION [ IF EXISTS ] qname ... DROP FILTER FUNCTION [ IF EXISTS ] qname ... DROP LOADER [ IF EXISTS ] qname ... DROP PROCEDURE [ IF EXISTS ] qname ... DROP TRIGGER [ IF EXISTS ] qname ... * Lots of changes to streamline the internal error handling. In principle, all errors should now include a SQLSTATE error code (see the SQL standard). * Added support for extracting the quarter (number between 1 and 4) of a date or a timestamp or a timestamp with timezone in SQL: EXTRACT ( QUARTER FROM my_date_expr ). Added support for extracting the week (number between 1 and 53) of a date or a timestamp or a timestamp with timezone in SQL: EXTRACT ( WEEK FROM my_date_expr ). Added support for scalar functions: quarter(date_expr), quarter(timestamp_expr) and quarter(timestamptz_expr). Bug Fixes * 3574: Add support for: create OR REPLACE view ... * 3824: Created table not visible from ODBC * 3831: Extend date part extraction and date formating functions to support more formats like "quarter", "iso year/week" * 6244: Add support for: TRUNCATE TABLE * 6249: DEFAULT in row-values missing (sqlsmith) * 6346: BATsort returns GDK_SUCCEED when **sorted bat is null * 6438: Implement functionality to enforce the restrict option in: DROP SCHEMA xyz RESTRICT; * 6507: Column Header coming with trailing spaces in compiled version of MonetDB 11.27.9/11 * 6513: Sqlitelogictest: Wrong MAL plan generation for column product * 6526: Crash using aggregate function inside a case statement in having clause * 6529: Sqlitelogictest crash in select query with IN operator and cast * 6530: Sqlitelogictest: select query with NOT IN giving wrong results * 6532: copy into ignore null as directive if first column doesn't come from file * 6534: [Mar2018]: mclient -f tab / --format=tab complains about "unsupported formatter" * 6535: [Mar2018]: mclient -t / --timer does not work as documented * 6536: [Mar2018]: timing output of mclient -t / --timer= should go to stderr rather than stdout * 6537: [Mar2018]: mclient's default timing mode should (again) be "none" rather than "clock" * 6541: [Mar2018]: mclient reports incorrect wall-clock time * 6542: assertion failure when querying: select count(*) from sys.commented_function_signatures; * 6543: Mar2018: truncate on SQL system tables should NOT be allowed * 6545: Sqlitelogictest crash in IN query * 6546: Sqlitelogictest crash in IN query with division * 6547: OS-dependent behaviour for ilike * 6548: Select from remote table leaves session open * 6549: Add log messages when listen fails * 6550: Sqlitelogictest crash on complex CASE statement * 6551: Sqlitelogictest wrong NULL value cast * 6552: Sqlitelogictest crash on complex case statement * 6553: Sqlitelogictest crash on aggregation with having statement * 6554: Sqlitelogictest crash on complex case statement * 6555: Sqlitelogictest wrong result set generated from complex case statement * 6556: Sqlitelogictest division by zero on COALESCE call * 6557: Sqlitelogictest crash on aggregation query with not in * 6559: rows in sys.statistics are not removed when a temporary table is dropped. * 6560: Sqlitelogictest crash on group by query with having in * 6561: Sqlitelogictest crash on group by query with having not in clause

1 0

tail heap hash found to be invalid on repeated DB loads
by Eyal Rozenberg 29 Mar '18

29 Mar '18

As part of, shall we say, helping with the QA of the March-feature-branch-based releases, I'm looking at some of the GDK code involving loading a BBP (and the action thereof). Specifically, I've been looking at the function gdk/gdk_atoms.c:strCleanHash(), which recomputes the opportunistic partial hash table for a tail heap; checks if it's equal to what (seems to be) the hash loaded from disk; and if they don't match - copies the new hash over the old one. I've added an indication in case the hashes don't match. And - I've noticed this happens repeatedly when trying to load the same database, even after previous plain vanilla unmodified source code runs, with no crashes and orderly shutdown. At least - I'm seeing this happen with 11.29.1, and on a TPC-H SF 1 database I created with the script at https://github.com/eyalroz/tpch-tools My questions: 1. Is a discrepancy between the old and recomputed hash, when there had not been any problems on the previous run, expected behavior - or is it a bug? 2. If it is expected, why isn't a correct hash written back to the disk? Doesn't the memcmp(newhash, h->base, sizeof(newhash)) command eventually/immediately get translated into a write? 3. If this is not expected behavior - has this been verified not to happen on other machines? Other versions? 4. Could this be a 11.29.3 vs 11.29.1 issue somehow? I haven't updated to the latest RC yet. Eyal

2 2

Getting the number of BAT accesses
by Vsevolod Sevostyanov 20 Feb '18

20 Feb '18

Dear developers of MonetDB, I am trying to measure the number of BAT accesses by computing per each BAT a sum of differences between highest and lowest indexes in ALGsubselect2. I have instrumented the BAT descriptor with an additional integer value and added its initialization into BATcreate. I also added an increment operation of the value into ALGsubselect2 and modified the COLcopy function in order to carry the acquired number around. But it all seems to be in vain, all the values remain equal to 0 when I check them via code. Is my approach even sensible? Where else should I look to make this work? I use MonetDB/RDF branch, the latest available version (monetdbd 1.7, monetdb 1.1). Thank you in advance! Best Regards, Vsevolod

1 0

Announcement: New Jul2017-SP4 Bugfix release of MonetDB suite
by Sjoerd Mullender 12 Feb '18

12 Feb '18

The MonetDB team at CWI/MonetDB BV is pleased to announce the Jul2017-SP4 bugfix release of the MonetDB suite of programs. More information about MonetDB can be found on our website at <http://www.monetdb.org/>. For details on this release, please see the release notes at <http://www.monetdb.org/Downloads/ReleaseNotes>. As usual, the download location is <http://dev.monetdb.org/downloads/>. Jul 2017-SP4 bugfix release (11.27.13) Build Environment * Added the .lib and .h files needed for building extensions to the Windows installer. Bug Fixes * 3470: Support setClob without length restrictions * 6468: JDBC 2.27 fails with year < 1000 * 6482: Query failures on order by on union * 6483: Monetdb crashes, on query * 6487: UNION of NULLs with several tables fails * 6488: Semijoin returns duplicate values from a column with unique values * 6489: Sqlitelogictest - Wrong result set of complex conditional query * 6490: Sqlitelogictest - Select query with an IN clause parse error * 6491: SELECT IN returns NULL instead of false when not found * 6492: Persistent hashes stored and then ignored. Storage info not in sync with actual indices. * 6493: Sqlitelogictest - Aggregation query on empty table with DISTINCT clause * 6494: Sqlitelogictest - Algebra operators priority in select query * 6495: Sqlitelogictest - Omitting AS in a result set column alias name * 6496: Sqlitelogictest - Select interval comparisons between floating-points and NULL * 6497: Sqlitelogictest - Select not between query producing wrong results * 6498: large virtual memory spike on BLOB column COUNT * 6499: Crash when trying to replace a function defined in sys from a different schema * 6502: Query with multiple limit clauses does not return anything * 6508: Segmentation fault in mserver5 on Python2 UDF with TIMESTAMP column input that has NULL values (conversion.c:438, PyNullMask_FromBAT) * 6510: Sqlitelogictest: Wrong output in aggregation query * 6512: Monetdb crashes on query with limit after sort with case * 6514: Sqlitelogictest: Range query between NULL values not possible * 6515: Insert null second interval value results in 0 * 6516: Sqlitelogictest unknown bat append operation * 6517: Sqlitelogictest overflow in conversion during MAL plan execution * 6518: Sqlitelogictest: count aggregation with not in operator * 6519: Sqlitelogictest: algebra join between lng and int BATs undefined * 6520: UPDATE with correlated subquery causes assertion (or segfault) * 6522: Sqlitelogictest: IN operator return a single column * 6523: Sqlitelogictest: Case statement subquery missing * 6524: Sqlitelogictest: Crash in aggregation query with IN operator * 6527: Crash using order by alias in subquery

1 0

New Mar2018 stable branch created
by Sjoerd Mullender 06 Feb '18

06 Feb '18

I have created a new stable branch for the next feature release. This branch was created in order to stabilize the code before the feature release. From now on *only* bug fixes may go to this new stable branch. I reserve the right to undo any changes that do not comply with this rule. I also reserve the right to admit changes that are not strictly bug fixes. The name for the branch is Mar2018. So in each of your stable checkout directories you can do hg pull hg update -rMar2018 to update to this new branch. -- Sjoerd Mullender

1 0

Re: MonetDB: comment-on - Use separate SQLstatementIntern to SET sys...
by Sjoerd Mullender 12 Jan '18

12 Jan '18

On 12/01/18 14:55, Joeri van Ruth wrote: > Changeset: ae04d7842803 for MonetDB > URL: https://dev.monetdb.org/hg/MonetDB?cmd=changeset;node=ae04d7842803 > Modified Files: > sql/backends/monet5/sql_upgrades.c > Branch: comment-on > Log Message: > > Use separate SQLstatementIntern to SET sys.keywords READ ONLY > > Hopefully this fixes the upgrade failures. It doesn't. :-( > > > diffs (45 lines): > > diff --git a/sql/backends/monet5/sql_upgrades.c b/sql/backends/monet5/sql_upgrades.c > --- a/sql/backends/monet5/sql_upgrades.c > +++ b/sql/backends/monet5/sql_upgrades.c > @@ -1070,12 +1070,12 @@ sql_remove_environment_func(Client c, mv > static str > sql_create_comments_table(Client c) > { > - char *err, *q1, *q2; > + char *err, *q; > > - q1 = "" > + q = "" > "ALTER TABLE sys.keywords SET READ WRITE;\n" > "INSERT INTO sys.keywords VALUES ('COMMENT');\n" > - "ALTER TABLE sys.keywords SET READ ONLY;\n" > + "-- ALTER TABLE sys.keywords SET READ ONLY;\n" > "\n" > "CREATE TABLE sys.comments (\n" > " id INTEGER NOT NULL PRIMARY KEY,\n" > @@ -1223,16 +1223,22 @@ sql_create_comments_table(Client c) > "FROM commented_function_params\n" > "ORDER BY line;\n" > "GRANT SELECT ON sys.commented_function_signatures TO PUBLIC;\n"; > - err = SQLstatementIntern(c, &q1, "update", 1, 0, NULL); > + err = SQLstatementIntern(c, &q, "update", 1, 0, NULL); > if (err) > return err; > > - q2 = "" > + q = "" > "UPDATE sys._tables\n" > "SET system = true\n" > "WHERE name = 'comments'\n" > "AND schema_id = (SELECT id FROM sys.schemas WHERE name = 'sys');\n"; > - return SQLstatementIntern(c, &q2, "update", 1, 0, NULL); > + > + err = SQLstatementIntern(c, &q, "update", 1, 0, NULL); > + if (err) > + return err; > + > + q = "ALTER TABLE sys.keywords SET READ ONLY;"; > + return SQLstatementIntern(c, &q, "update", 1, 0, NULL); > } > > void > _______________________________________________ > checkin-list mailing list > checkin-list(a)monetdb.org > https://www.monetdb.org/mailman/listinfo/checkin-list > -- Sjoerd Mullender

2 1

Getting the number of BAT accesses
by Vsevolod Sevostyanov 22 Dec '17

22 Dec '17

Dear developers of MonetDB, I am trying to measure the number of BAT accesses by computing per each BAT a sum of differences between highest and lowest indexes in ALGsubselect2. I have instrumented the BAT descriptor with an additional integer value and added its initialization into BATcreate. I also added an increment operation of the value into ALGsubselect2 and modified the COLcopy function in order to carry the acquired number around. But it all seems to be in vain, all the values remain equal to 0 when I check them via code. Is my approach even sensible? Where else should I look to make this work? I use MonetDB/RDF branch, the latest available version (monetdbd 1.7, monetdb 1.1). Thank you in advance! Best Regards, Vsevolod

1 0

Announcement: New Jul2017-SP3 Bugfix release of MonetDB suite
by Sjoerd Mullender 14 Dec '17

14 Dec '17

The MonetDB team at CWI/MonetDB BV is pleased to announce the Jul2017-SP3 bugfix release of the MonetDB suite of programs. More information about MonetDB can be found on our website at <http://www.monetdb.org/>. For details on this release, please see the release notes at <http://www.monetdb.org/Downloads/ReleaseNotes>. As usual, the download location is <http://dev.monetdb.org/downloads/>. Jul 2017-SP3 bugfix release (11.27.11) MonetDB Common * Reimplemented summing of a column of floating point (flt and dbl) values. The old code could give wildly inaccurate results when adding up lots and lots of values due to lack of precision. Try SELECT sum(c) FROM t; where t is 100,000,000 rows, c is of type REAL and all values are equal to 1.1. (The old code returned 33554432 instead of 1.1e8.) Bug Fixes * 3898: Deadlock on insertion * 6429: ROUND produces wrong data type * 6436: Query sequence with 2x ifthenelse() and next nullif() causes mserver5 Segmentation fault * 6437: System schemas "profiler" and "json" shouldn't be allowed to be dropped. * 6439: Invalid references to sys.columns.id from sys.statistics.column_id * 6442: SEGFAULT with COPY INTO BEST EFFORT and skipping input columns * 6443: complex(?) query forgets(?) column name * 6444: Using 'with' keyword with table returning function crashes monetdb * 6445: Sqlitelogictest crash in MySQL query * 6446: sql_parser.y bug? * 6448: 'insert into' with multiple rows containing subqueries crashes * 6449: Assertion error in rel_dce_refs (sqlsmith) * 6450: Assertion error in exp_bin (sqlsmith) * 6451: Assertion error in sql_ref_dec (sqlsmith) * 6453: Assertion error in rel_rename_exps (sqlsmith) * 6454: SQL lexical error * 6455: Assertion error in rel_apply_rewrite (sqlsmith) * 6456: NULL becomes 0 in outer join * 6459: Assertion error in exp_bin (sqlsmith) * 6462: large virtual memory spike on BLOB column select * 6465: appending to variables sized atom bats other than str bats with force flag may result in corrupted heap * 6467: date_to_str formatter is wrong * 6470: mitosis gets in the way of simple select * 6471: calls to sys.generate_series should auto-convert arguments * 6472: Assertion failure in rel_rename (Sqlsmith) * 6477: assertion eror rel_push_project_up (sqlsmith) * 6478: Crash with nested order by/ limit offset * 6479: Mserver receives an assertion error on a procedure call * 6480: Segfault in mvc_find_subexp (sqlsmith)

1 0

dbplot for R
by Hannes Mühleisen 30 Nov '17

30 Nov '17

Hi all, There is a new R package, dbplot [1], which pushes aggregations needed vor viz into the database. This partially works fine with MonetDB[Lite], but most examples [2] fail due to the lack of window aggregate support (min etc.). So perhaps this is a good reason to implement those. Hint-hint, Solutions. Hannes [1] https://github.com/edgararuiz/dbplot <https://github.com/edgararuiz/dbplot> [2] https://gist.github.com/hannesmuehleisen/d18a0f2f494db69e157abd753d6bf31d <https://gist.github.com/hannesmuehleisen/d18a0f2f494db69e157abd753d6bf31d>

1 0

Re: read BLOB column on tail heap and loop over rows
by Anton Kravchenko 13 Nov '17

13 Nov '17

Ok, here is a workaround [though I suspect that performance is going to degrade (with respect to CHAR column for example[1]) because BUNtail is being called for each row...] sql_column *col = mvc_bind_column(m, tbl, colname[icol]); BAT *b = table_funcs.full_column(m->session->tr, col); //read the entire column into continuous memory on the tail heap of the BAT b BATiter *iterators = NULL; blob *blob_record; int ib; iterators = GDKzalloc(sizeof(BATiter)); iterators[0] = bat_iterator(b); // blob column iterator for (irow=0; irow < nrow; irow++) { blob_record = (blob*) BUNtail(iterators[0], irow); for (ib=0; ib < blob_record->nitems; ib++) { printf("blob data = %c, %02x\n", blob_record->data[ib], blob_record->data[ib]); } } p.s. [1] //For CHAR column it would look like this: sql_column *col = mvc_bind_column(m, tbl, colname[icol]); BAT *b = table_funcs.full_column(m->session->tr, col); //read the entire column into continuous memory on the tail heap of the BAT b char *tvbase, *cptr; unsigned char *readucrows; tvbase = (char *) b->T.vheap->base + GDK_VAROFFSET; readucrows = (unsigned char *) b->T.heap.base; for (irow=0; irow < nrow; irow++) { cptr = tvbase + *readucrows++; } Anton On Fri, Nov 10, 2017 at 4:13 PM, Anton Kravchenko < kravchenko.anton86(a)gmail.com> wrote: > I wonder how one would read BLOB column and loop over its rows? > > BAT *b = table_funcs.full_column(m->session->tr, blob_col_name); > //blob *readblobrows = (blob*) b->T.heap.base; > //unsigned char *readblobrows = (unsigned char*) b->T.heap.base; > > p.s. > For INT column it would be go like this: > int * readintrows = (int *) b->T.heap.base; > for (irow=0; irow < nrow; irow++) { > intvalue = *readintrows++; > } > > Thanks, > Anton >

1 0

Re: MonetDB: Jul2017 - Properly respect DEFAULT values in the Python...
by Stefan Manegold 13 Nov '17

13 Nov '17

Mark, thanks for taking care of this!! Unfortunately, it does not seem to work correctly, yet. For strings, as in your pyloader01.sql test (see below), the used/inserted default value appears to include the single-quotes "'" form the SQL syntax --- I assume you approved those by accident --- and for integers "strange values" are inserted; see test sql/test/BugTracker-2017/Tests/python_loader_clobbers_default_with_null.Bug-6464.sql Best, Stefan ----- On Nov 13, 2017, at 5:47 PM, Mark Raasveldt commits+m.raasveldt=cwi.nl(a)monetdb.org wrote: > Changeset: 4303f7489624 for MonetDB > URL: https://dev.monetdb.org/hg/MonetDB?cmd=changeset;node=4303f7489624 > Modified Files: > sql/backends/monet5/Tests/pyloader01.sql > sql/backends/monet5/Tests/pyloader01.stable.out > sql/backends/monet5/UDF/pyapi/emit.c > sql/backends/monet5/UDF/pyapi/pyloader.c > sql/include/sql_catalog.h > sql/server/rel_updates.c > Branch: Jul2017 > Log Message: > > Properly respect DEFAULT values in the Python loader. > > > diffs (145 lines): > > diff --git a/sql/backends/monet5/Tests/pyloader01.sql > b/sql/backends/monet5/Tests/pyloader01.sql > --- a/sql/backends/monet5/Tests/pyloader01.sql > +++ b/sql/backends/monet5/Tests/pyloader01.sql > @@ -1,7 +1,7 @@ > > START TRANSACTION; > > -CREATE TABLE mytable(a DOUBLE, d int, s STRING); > +CREATE TABLE mytable(a DOUBLE, d int, s STRING DEFAULT 'hello'); > > CREATE LOADER myfunc() LANGUAGE PYTHON { > _emit.emit({'a':42,'d':1}) > @@ -23,10 +23,10 @@ SELECT name,func,mod,language,type,side_ > > > -- there is a reason for this, functions with 0, 1, 2 and 3+ arguments are > handled differently. > +COPY LOADER INTO mytable FROM myfunc(); > COPY LOADER INTO mytable FROM myfunc3(46, 'asdf', 3.2); > COPY LOADER INTO mytable FROM myfunc2(45, 'asdf'); > COPY LOADER INTO mytable FROM myfunc1(44); > -COPY LOADER INTO mytable FROM myfunc(); > > SELECT * FROM mytable; > > diff --git a/sql/backends/monet5/Tests/pyloader01.stable.out > b/sql/backends/monet5/Tests/pyloader01.stable.out > --- a/sql/backends/monet5/Tests/pyloader01.stable.out > +++ b/sql/backends/monet5/Tests/pyloader01.stable.out > @@ -56,11 +56,11 @@ Ready. > % sys.mytable, sys.mytable, sys.mytable # table_name > % a, d, s # name > % double, int, clob # type > -% 24, 1, 5 # length > +% 24, 1, 7 # length > +[ 42, 1, "'hello'" ] > [ 46, 4, "hello" ] > -[ 45, 3, NULL ] > -[ 44, 2, NULL ] > -[ 42, 1, NULL ] > +[ 45, 3, "'hello'" ] > +[ 44, 2, "'hello'" ] > #DROP TABLE mytable; > #DROP ALL LOADER myfunc; > #CREATE LOADER myfunc() LANGUAGE PYTHON { > diff --git a/sql/backends/monet5/UDF/pyapi/emit.c > b/sql/backends/monet5/UDF/pyapi/emit.c > --- a/sql/backends/monet5/UDF/pyapi/emit.c > +++ b/sql/backends/monet5/UDF/pyapi/emit.c > @@ -199,6 +199,7 @@ PyObject *PyEmit_Emit(PyEmitObject *self > > self->cols[self->ncols].b = COLnew(0, bat_type, 0, TRANSIENT); > self->cols[self->ncols].name = GDKstrdup(val); > + self->cols[self->ncols].def = NULL; > if (self->nvals > 0) { > // insert NULL values up until the current entry > for (ai = 0; ai < self->nvals; ai++) { > @@ -209,9 +210,9 @@ PyObject *PyEmit_Emit(PyEmitObject *self > goto wrapup; > } > } > - self->cols[i].b->tnil = 1; > - self->cols[i].b->tnonil = 0; > - BATsetcount(self->cols[i].b, self->nvals); > + self->cols[self->ncols].b->tnil = 1; > + self->cols[self->ncols].b->tnonil = 0; > + BATsetcount(self->cols[self->ncols].b, self->nvals); > } > self->ncols++; > } > @@ -337,14 +338,21 @@ PyObject *PyEmit_Emit(PyEmitObject *self > self->cols[i].b->tnonil = 1 - self->cols[i].b->tnil; > } > } else { > + void* nill_value = ATOMnil(self->cols[i].b->ttype); > + void* default_value = self->cols[i].def ? > + self->cols[i].def : > + nill_value; > for (ai = 0; ai < (size_t)el_count; ai++) { > - if (BUNappend(self->cols[i].b, ATOMnil(self->cols[i].b->ttype), > + if (BUNappend(self->cols[i].b, > + default_value, > 0) != GDK_SUCCEED) { > goto wrapup; > } > } > - self->cols[i].b->tnil = 1; > - self->cols[i].b->tnonil = 0; > + if (BATatoms[self->cols[i].b->ttype].atomCmp(default_value, nill_value) == > 0) { > + self->cols[i].b->tnil = 1; > + self->cols[i].b->tnonil = 0; > + } > } > BATsetcount(self->cols[i].b, self->nvals + el_count); > } > diff --git a/sql/backends/monet5/UDF/pyapi/pyloader.c > b/sql/backends/monet5/UDF/pyapi/pyloader.c > --- a/sql/backends/monet5/UDF/pyapi/pyloader.c > +++ b/sql/backends/monet5/UDF/pyapi/pyloader.c > @@ -146,6 +146,9 @@ PYFUNCNAME(PyAPIevalLoader)(Client cntxt > assert(i < pci->retc); > cols[i].name = GDKstrdup(*((char **)n->data)); > n = n->next; > + assert(n); > + cols[i].def = n->data; > + n = n->next; > cols[i].b = > COLnew(0, getBatType(getArgType(mb, pci, i)), 0, TRANSIENT); > cols[i].b->tnil = 0; > diff --git a/sql/include/sql_catalog.h b/sql/include/sql_catalog.h > --- a/sql/include/sql_catalog.h > +++ b/sql/include/sql_catalog.h > @@ -599,6 +599,7 @@ extern node *find_sql_func_node(sql_sche > typedef struct { > BAT *b; > char* name; > + void* def; > } sql_emit_col; > > #endif /* SQL_CATALOG_H */ > diff --git a/sql/server/rel_updates.c b/sql/server/rel_updates.c > --- a/sql/server/rel_updates.c > +++ b/sql/server/rel_updates.c > @@ -1173,7 +1173,7 @@ table_column_types(sql_allocator *sa, sq > } > > static list * > -table_column_names(sql_allocator *sa, sql_table *t) > +table_column_names_and_defaults(sql_allocator *sa, sql_table *t) > { > node *n; > list *types = sa_list(sa); > @@ -1181,6 +1181,7 @@ table_column_names(sql_allocator *sa, sq > if (t->columns.set) for (n = t->columns.set->h; n; n = n->next) { > sql_column *c = n->data; > append(types, &c->base.name); > + append(types, c->def); > } > return types; > } > @@ -1577,7 +1578,7 @@ copyfromloader(mvc *sql, dlist *qname, s > return NULL; > } > ((sql_subfunc*) import->f)->res = table_column_types(sql->sa, t); > - ((sql_subfunc*) import->f)->colnames = table_column_names(sql->sa, t); > + ((sql_subfunc*) import->f)->colnames = > table_column_names_and_defaults(sql->sa, t); > > for (n = t->columns.set->h; n; n = n->next) { > sql_column *c = n->data; > _______________________________________________ > checkin-list mailing list > checkin-list(a)monetdb.org > https://www.monetdb.org/mailman/listinfo/checkin-list -- | Stefan.Manegold(a)CWI.nl | DB Architectures (DA) | | www.CWI.nl/~manegold/ | Science Park 123 (L321) | | +31 (0)20 592-4212 | 1098 XG Amsterdam (NL) |

1 0

MonetDB Integration Question
by David Broneske 13 Nov '17

13 Nov '17

Dear developers of MonetDB, I hope you are the right recipients of this mail that can help us in getting some more insights into MonetDB’s Code. Currently, me and my student (Florian) want to integrate our index structure Elf (www.elf.ovgu.de <http://www.elf.ovgu.de/>) into MonetDB as a proof of concept and to show its competitiveness to fast main-memory scans. However, we’ve got a hard time to understand some part of the code — we hope that you could help us with this. While the kernel (gdk) is well documented, we hardly find any documentation that may help us in answering the following questions: Which function takes care of filling the BATs when inserting into a table (i.e., sql_table/sql_rel)? COPY INTO creates a MAL-procedure copy_into, which is preparing the copy, but does not finally execute the copy. Which function is executed for this? Regarding ChangeSets: what is dset and nelm used for? Any pointer or explanation for these questions would be great! Regards, David and Florian M.Sc. David Broneske Otto-von-Guericke-Universität Magdeburg Department of Technical and Business Information Systems <http://wwwiti.cs.uni-magdeburg.de/> Databases and Software Engineering <http://wwwiti.cs.uni-magdeburg.de/iti_db/index_en.php> P.O. Box 4120, D-39016 Magdeburg, Germany Office: Building 29, Room112 Tel: +49 391 67-52994 <tel:+49 391 67-52994> Fax: +49 391 67-12020 <fax:+49 391 67-12020>

2 1

Announcement: New Jul2017-SP2 Bugfix release of MonetDB suite
by Sjoerd Mullender 26 Oct '17

26 Oct '17

The MonetDB team at CWI/MonetDB BV is pleased to announce the Jul2017-SP2 bugfix release of the MonetDB suite of programs. More information about MonetDB can be found on our website at <http://www.monetdb.org/>. For details on this release, please see the release notes at <http://www.monetdb.org/Downloads/ReleaseNotes>. As usual, the download location is <http://dev.monetdb.org/downloads/>. Jul 2017-SP2 bugfix release (11.27.9) MonetDB Common * A serious bug, possibly resulting in database corruption, having to do with appending data to a string BAT was fixed. Bug Fixes * 4017: server crashes when executing particular loopback query in embedded python * 6207: identifier ambiguous when grouping and selecting the same column twice * 6239: Incorrect profiling * 6261: New handling of delta tables hurts badly reusage of bats * 6287: should the CORR function return some numeric type that allows fractions? * 6321: Two-column aggregation on join result extremely slow. * 6335: Sqlitelogictest crash in complex SQL query * 6343: MERGE TABLE issue: unable to find * 6348: Interference of procedure/table name * 6350: Carriage return and form feed in TEXT fields are returned as 'r' and 'f' in jdbcclient and ResultSets * 6352: Scope resolution problem (sqlsmith) * 6353: implicit NULL value not propagated in distributed/remote query * 6374: Wrong answer from merge table after content changes * 6379: Table UDF: SEGV raised when invoking a non existing function * 6380: unable to create new databases from clean installation * 6381: Parser misses error messages in conditional * 6382: Can't set JSON fields via PreparedStatement * 6384: crash when setting a wrong listenaddr * 6385: AGGREGATE UDFs with more than 2 parameters incorrectly processed * 6386: Unexpected error from server for query with long floats * 6387: Performance degradation on multi column sort * 6388: JDBC Connection via user voc produces errors when fetching certain meta data information * 6392: SELECT EXISTS (empty table) returns 'true' * 6395: BAT leak of scalar result sets * 6397: Isolation of generating functions not correct * 6398: Null Matches in outer join are not supported * 6399: UDF crashes when subquery and scalar values are passed as pameters * 6400: getCharacterStream() currently not supported * 6404: COPY INTO crashes if table has primary key or foreign key constraint * 6405: Compilation error if DISABLE_PARENT_HASH not defined * 6409: sqllogictest crash on aggregation query with NOT IN clause in HAVING clause * 6410: Sqlitelogictest crash on aggregation query with IN clause * 6411: Sqlitelogictest crash in aggregation query * 6412: Overflow in sys.epoch * 6413: Wrong answer for TPC-H Q17 * 6414: Using RPAD returns: could not allocate space * 6416: Sqlitelogictest crash in aggregation query with a NOT IN clause * 6417: Segfault encountered (sqlsmith) * 6418: Segfault in renaming (sqlsmith) * 6419: segfault in rel_optimizer (sqlsmith) * 6420: Assertion error in mergetable task (sqlsmith) * 6422: Another assertion error in rel_or (sqlsmith) * 6423: Dereference null pointer (sqlsmith) * 6424: Assertion error in rel_rename_expr (sqlsmith) * 6425: Assertion error in exp_bin (sqlsmith) * 6426: Assertion error in rel_find_exp_ (sqlsmith) * 6427: Assertion error in eq_typeswitchloop (sqlsmith) * 6428: Sqlitelogictest crash in aggregation query * 6430: Assertion raised in another eq_typeswitch error (sqlsmith) * 6431: Sqlitelogictest crash in aggregation query with a long having clause * 6432: Assertion error in exp_bin (sqlsmith) * 6433: Sqlitelogictest crash in complex SELECT query with IN operator * 6435: Sqlitelogictest crash in simple select query * 6437: System schemas "profiler" and "json" shouldn't be allowed to be dropped. * 6438: Implement functionality to enforce the restrict option in: DROP SCHEMA xyz RESTRICT; * 6440: Faulty plan generated. Query returns more rows than expected or existing in the view sys.tables.

1 0

IMPORTANT: serious possibility for database corruption
by Sjoerd Mullender 23 Oct '17

23 Oct '17

We just found (and fixed) a serious problem in the Jul2017 releases that can result in database corruption. So far we have only seen the corruption occur after a database (mserver5) restart. We have not yet seen a problem with a server that continues running (but unfortunately no guarantees). We will create a new release candidate with this bug fixed as soon as possible (that will be version 11.27.9, i.e. anything in the 11.27.X series older than this is affected). Currently known work around is to *not* restart the server, but keep it running, and creating a dump of the database using msqldump. This dump will then have to be restored into a fresh database in a server that is not affected by the bug (Dec2016 release or the upcoming Jul2017-SP2). -- Sjoerd Mullender

1 0

Announcement: New Jul2017-SP1 Bugfix release of MonetDB suite
by Sjoerd Mullender 02 Aug '17

02 Aug '17

The MonetDB team at CWI/MonetDB BV is pleased to announce the Jul2017-SP1 bugfix release of the MonetDB suite of programs. More information about MonetDB can be found on our website at <http://www.monetdb.org/>. For details on this release, please see the release notes at <http://www.monetdb.org/Downloads/ReleaseNotes>. As usual, the download location is <http://dev.monetdb.org/downloads/>. Jul 2017-SP1 bugfix release Build Environment * The Debian and Ubuntu installers have been fixed: there was a file missing in the Jul2017 release. * Added a new RPM called MonetDB-selinux which provides the SELinux policy required to run MonetDB under systemd, especially on Fedora 26. * The Windows installers (*.msi files) are now created using the WiX Toolset. * The Windows binaries are now built using Visual Studio 2015. Because of this, you may need to install the Visual C++ Redistributable for Visual Studio 2015 before being able to run MonetDB. Merovingian * monetdbd was leaking open file descriptors to the mserver5 process it started. This has been fixed. MonetDB Common * Many functions in GDK are now annotated with the GCC attribute __warn_unused_result__ meaning that the compiler will issue a warning if the result of the function (usually an indication of an error) is not used. Bug Fixes * 6325: Merge table unusable in other connections * 6328: Transactional/multi-connection issues with merge tables * 6336: VALUES multiple inserts error * 6339: Mserver5 crashes on nested SELECT * 6340: sample operator takes effect after the execution of the query, expected before * 6341: MERGE TABLE issue: Cannot register * 6342: MERGE TABLE issue: hang * 6344: Spurious errors and assertions (SQLsmith) * 6375: MAL profiler truncates JSON objects larger than 8192 characters

1 0

Trying to make sense of multi-farm databases
by Eyal Rozenberg 21 Jul '17

21 Jul '17

Hello all, I've noticed a change introduced to the release version in Jun 2016 that I can't find a bugzilla bug or a detailed comment for, and I wish to understand a little better: multi-farm DBs, or multi-farm BAT buffer pools if you will. Questions: * Why do we have them? * Are BAT "roles", introduced with the patch for this issue, inherently relevant only with multiple DB farms? If not, why were they introduced at the same time and with the same changeset? * Is this considered a "good idea" design-wise, or perhaps an unfortunate necessity (e.g. due to client needs)? * When you create a new DB, is there a chance more than a single DB farm will be created when you haven't specifically requested this happen? Eyal

2 2

Announcement: New Jul2017 Feature release of MonetDB suite
by Sjoerd Mullender 18 Jul '17

18 Jul '17

The MonetDB team at CWI/MonetDB BV is pleased to announce the Jul2017 feature release of the MonetDB suite of programs. More information about MonetDB can be found on our website at <http://www.monetdb.org/>. For details on this release, please see the release notes at <http://www.monetdb.org/Downloads/ReleaseNotes>. As usual, the download location is <http://dev.monetdb.org/downloads/>. Jul 2017 feature release MonetDB5 Server * The "sub" prefix of many functions, both at the MAL and the C level, has been removed. * Changed the interfaces of the AUTH* functions: pass values, not pointers to values. * Removed calc.setoid(). * group.subgroup is now called group.group if it is not refining a group. Both group.group and group.subgroup now also have variants with a candidate list. * The allocation schemes for MAL blocks and Variables has been turned into block-based. This reduces the number of malloc()/free() calls. . * Added a new server-side protocol implementation. The new protocol is backwards compatible with the old protocol. Clients can choose whether they want to use the old or the new protocol during the initial handshake with the server. The new protocol is a binary column-based protocol that is significantly faster than the old protocol when transferring large result sets. In addition, the new protocol supports compression using Snappy or LZ4. * Moved the sphinx extension module to its own repository. See https://dev.monetdb.org/hg/MonetDB-sphinx/. * Removed GSL module: it's now a separate (extension) package. See https://dev.monetdb.org/hg/MonetDB-gsl/. * The PCRE library is now optional for systems that support POSIX regular expressions. * Added 5 new sys schema tables: function_languages, function_types, key_types, index_types and privilege_codes. They are pre-loaded with static content and contain descriptive names for the various integer type and code values. See also sql/scripts/51_sys_schema_extension.sql Merovingian * Added handling of a dbextra property per database at the daemon level. The user can set the dbextra property for a database using the command: $ monetdb set dbextra=<path> <database> and the daemon will make sure to start the new server using the correct --dbextra parameter. Client Package * The mclient and msqldump programs lost compatibility with old mserver5 versions (pre 2014) which didn't have a "system" column in the sys.schemas table. * The mclient and msqldump programs lost compatibility with ancient mserver5 versions (pre 2011) which didn't have the sys.systemfunctions table. * Removed the "array" and "quick" functions from the mapi library. To be precise, the removed functions are: mapi_execute_array, mapi_fetch_field_array, mapi_prepare_array, mapi_query_array, mapi_quick_query, mapi_quick_query_array, and mapi_quick_response. * Added a more elaborate \help command for SQL expressions. MonetDB Common * Improved error checking in the logger code (dealing with the write-ahead log); changed return types a several functions from int to gdk_return (i.e., they now return GDK_SUCCEED or GDK_FAIL). The logger no longer calls GDKfatal on error. Instead the caller is responsible for dealing with errors. * BATsort may now create an order index as a by product. * Quantile calculations now use the order index if available (and use BATsort otherwise, producing an order index). * Quantiles calculate a position in the sorted column. If this position is not an integer, we now choose the nearest position, favoring the lower if the distance to the two adjacent positions is equal (round down to nearest integer). * Removed function BATprintf. Use BATprint or BATprintcolumns instead. * Removed BATsave from the list of exported functions. * Replaced BBPincref/BBPdecref with BBPfix/BBPunfix for physical reference count and BBPretain/BBPrelease for logical reference count maintenance. * Removed automatic conversion of 32-bit OIDs to 64 bits on 64-bit architectures. * Removed functions OIDbase() and OIDnew(). * Removed talign field from BAT descriptor. * BATappend now takes an optional (NULL if not used) candidate list for the to-be-appended BAT. * New function BATkeyed(BAT *b) that determines (possibly using a hash table) whether all values in b are distinct. SQL * Made the operator precedence of % equal to those of * and /. All three are evaluated from left to right. * Removed table sys.connections. It was a remnant of an experimental change that had already been removed in 2012. * Protect against runaway profiler events If you hit a barrier block during profiling, the JSON event log may quickly become unwieldy. Event production is protected using a high water mark, which ensures that never within the single execution of MAL block the instruction causes excessive event records. Bug Fixes * 3465: Request: add support for CREATE VIEW with ORDER BY clause * 3545: monetdb commands don't work with -h -P -p options (locally and remotely) * 3996: select * from sys.connections always returns 0 rows. Expected to see at least one row for the active connection. * 6187: Nested WITH queries not supported * 6225: Order of evaluation of the modulo operator * 6289: Crashes and hangs with remote tables * 6292: Runaway SQL optimizer in too many nested operators * 6310: Name resolution error (sqlsmith) * 6312: Object not found in LIMIT clause (sqlsmith) * 6313: Null type resolution in disjunction fails (sqlsmith) * 6319: Server crash on LATERAL (sqlsmith) * 6322: Crash on disjunction with LIMIT (sqlsmith) * 6323: Deadlock calling sys.bbp() * 6324: Sqlitelogictest crash in a IN query (8th) * 6327: The daemon does not respect the actual name of the mserver5 executable * 6330: Sqlitelogictest crash on a complex SELECT query * 6331: sys.statistics column "nils" always contains 0. Expected a positive value for columns that have one or more nils/NULLs * 6332: Sqlitelogictest crash related to an undefined MAL function

3 3

Re: MonetDB: default - Support LIMIT in scalar expressions
by Lefteris 12 Jul '17

12 Jul '17

maybe we could also support FIRST() and LAST() https://www.monetdb.org/bugzilla/show_bug.cgi?id=6333 On Wed, Jul 12, 2017 at 11:22 AM, Martin Kersten <commits+mk=cwi.nl(a)monetdb.org> wrote: > Changeset: 718758b70621 for MonetDB > URL: https://dev.monetdb.org/hg/MonetDB?cmd=changeset;node=718758b70621 > Modified Files: > sql/server/sql_parser.y > Branch: default > Log Message: > > Support LIMIT in scalar expressions > Picking a single element from a table is now supported using > LIMIT (and SAMPLE) in SET and UPDATE subqueries > > set v =(select * from tbl limit 1); > > > diffs (21 lines): > > diff --git a/sql/server/sql_parser.y b/sql/server/sql_parser.y > --- a/sql/server/sql_parser.y > +++ b/sql/server/sql_parser.y > @@ -772,7 +772,7 @@ set_statement: > append_string(l, $2 ); > append_symbol(l, $4 ); > $$ = _symbol_create_list( SQL_SET, l); } > - | set column_commalist_parens '=' subquery > + | set column_commalist_parens '=' subquery_with_orderby > { dlist *l = L(); > append_list(l, $2); > append_symbol(l, $4); > @@ -2919,7 +2919,7 @@ assignment: > append_symbol(l, $3 ); > append_string(l, $1); > $$ = _symbol_create_list( SQL_ASSIGN, l); } > - | column_commalist_parens '=' subquery > + | column_commalist_parens '=' subquery_with_orderby > { dlist *l = L(); > append_symbol(l, $3); > append_list(l, $1); > _______________________________________________ > checkin-list mailing list > checkin-list(a)monetdb.org > https://www.monetdb.org/mailman/listinfo/checkin-list

2 2

Re: MonetDB: Jul2017 - Initialize sys.argv, because tensorflow requi...
by Panagiotis Koutsourakis 22 Jun '17

22 Jun '17

Hello developers, Taking the opportunity from this commit, I wanted to initiate a discussion with respect the executable names inside the code. A few weeks ago I faced a problem when using the configure options --program-prefix and --program-suffix, that change the executable names. The problem was that the name mserver5 was hard coded and the daemon could not find the actual executable. I see two possible solutions: 1. Implement a mechanism that make a note in the monetdb_config.h 2. Implement a run time mechanism to compute the relevant file names. Solution #1 is probably a bit more difficult because the prefix/suffix transformation of the file names happens during the installation. Solution #2 is already partially implemented in the daemon (see the function get_bin_path in the file mutils.c), but will probably need to be generalized somehow. Comments? Ideas? Best, Panos. Mark Raasveldt @ 2017-06-15 13:22 GMT: > Changeset: 44e0bc7dab59 for MonetDB > URL: http://dev.monetdb.org/hg/MonetDB?cmd=changeset;node=44e0bc7dab59 > Modified Files: > sql/backends/monet5/UDF/pyapi/pyapi.c > Branch: Jul2017 > Log Message: > > Initialize sys.argv, because tensorflow requires this. > > > diffs (20 lines): > > diff --git a/sql/backends/monet5/UDF/pyapi/pyapi.c b/sql/backends/monet5/UDF/pyapi/pyapi.c > --- a/sql/backends/monet5/UDF/pyapi/pyapi.c > +++ b/sql/backends/monet5/UDF/pyapi/pyapi.c > @@ -1425,9 +1425,16 @@ PYFUNCNAME(PyAPIprelude)(void *ret) { > MT_lock_init(&queryLock, "query_lock"); > MT_lock_set(&pyapiLock); > if (!pyapiInitialized) { > +#ifdef IS_PY3K > + wchar_t* program = Py_DecodeLocale("mserver5", NULL); > + wchar_t* argv[] = { program }; > +#else > + char* argv[] = {"mserver5"}; > +#endif > str msg = MAL_SUCCEED; > PyObject *tmp; > Py_Initialize(); > + PySys_SetArgvEx(1, argv, 0); > _import_array(); > msg = _connection_init(); > if (msg != MAL_SUCCEED) { > _______________________________________________ > checkin-list mailing list > checkin-list(a)monetdb.org > https://www.monetdb.org/mailman/listinfo/checkin-list

5 4

Announcement: New Dec2016-SP5 Bugfix release of MonetDB suite
by Sjoerd Mullender 12 Jun '17

12 Jun '17

The MonetDB team at CWI/MonetDB BV is pleased to announce the Dec2016-SP5 bugfix release of the MonetDB suite of programs. More information about MonetDB can be found on our website at <http://www.monetdb.org/>. For details on this release, please see the release notes at <http://www.monetdb.org/Downloads/ReleaseNotes>. As usual, the download location is <http://dev.monetdb.org/downloads/>. Dec 2016-SP5 bugfix release Bug Fixes * 6290: Crash (and assertion failure) with a correlated subquery with NOT IN in the select-list * 6291: crash if executes a function with sample operator * 6294: Sqlitelogictest crash * 6296: Another sqllitelogictest crash :( * 6297: 7th sqllitelogictest crash :( * 6300: Protect against missing BATs (sqlsmith) * 6314: Lateral crash report (sqlsmith) * 6315: Exist operator on type bigint missing (sqlsmith) * 6316: Coalesc and limit error (sqlsmith) * 6317: Two-column integer aggregation extremely slow * 6318: Daemon crashes if .merovignian_properties for a database contains more than 42 entries * 6320: The daemon ignores all but the last entry in the .merovignian_properties file when spawning mserver5

1 0

Re: MonetDB: Jul2017 - Daemon was sending wrong answer for authoriza...
by Roberto Cornacchia 01 Jun '17

01 Jun '17

Thanks! This was one of the longest awaited fixes. Funny that it was just about a newline. Perhaps the bugfix can go to Dec2016? Best, Roberto On 31 May 2017 at 13:55, Panagiotis Koutsourakis < commits+kutsurak=monetdbsolutions.com(a)monetdb.org> wrote: > Changeset: e00fb765660d for MonetDB > URL: http://dev.monetdb.org/hg/MonetDB?cmd=changeset;node=e00fb765660d > Modified Files: > tools/merovingian/daemon/controlrunner.c > Branch: Jul2017 > Log Message: > > Daemon was sending wrong answer for authorization request > > When the password was correct, the server would send one extra new line > that > confused the client. This commit fixes bug 3545 > > > diffs (12 lines): > > diff --git a/tools/merovingian/daemon/controlrunner.c > b/tools/merovingian/daemon/controlrunner.c > --- a/tools/merovingian/daemon/controlrunner.c > +++ b/tools/merovingian/daemon/controlrunner.c > @@ -169,7 +169,7 @@ control_authorise( > } > free(pwd); > > - mnstr_printf(fout, "=OK\n"); > + mnstr_printf(fout, "=OK"); > mnstr_flush(fout); > > return 1; > _______________________________________________ > checkin-list mailing list > checkin-list(a)monetdb.org > https://www.monetdb.org/mailman/listinfo/checkin-list >

2 2

Re: MonetDB: Dec2016 - fix for bug 6310, ie properly use the left pa...
by Martin Kersten 13 May '17

13 May '17

That's quick. I am using Jul2017 for testing sqlsmith please roll forward and I'll hunt for the next one. FYI on rocks006 /export/scratch1/home/mk/ssmonetdb you find the current output of ?160K queries (still running) grep ERROR output|sort -u gives the list that triggers the one I found. There are a few more to be investigated (after grocery shopping), e.g. ERROR = !Can not access column ERROR = !can not access index column ERROR = !Can not create object ERROR = !exist operator on type clob missing ERROR = !exist operator on type int missing ERROR = !exist operator on type varchar missing ERROR = !Illegal argument ERROR = !Object not found On 13/05/2017 11:00, Niels Nes wrote: > Changeset: fea5dccda850 for MonetDB > URL: http://dev.monetdb.org/hg/MonetDB?cmd=changeset;node=fea5dccda850 > Modified Files: > sql/server/rel_select.c > Branch: Dec2016 > Log Message: > > fix for bug 6310, ie properly use the left part of a lateral join, > when the left part isn't used in the right part. > > > diffs (29 lines): > > diff --git a/sql/server/rel_select.c b/sql/server/rel_select.c > --- a/sql/server/rel_select.c > +++ b/sql/server/rel_select.c > @@ -4902,7 +4902,7 @@ rel_query(mvc *sql, sql_rel *rel, symbol > sql_rel *fnd = NULL; > > for (n = fl->h; n ; n = n->next) { > - int lateral = check_is_lateral(n->data.sym); > + int lateral = check_is_lateral(n->data.sym), lateral_used = 0; > > fnd = table_ref(sql, NULL, n->data.sym, 0); > if (!fnd && (rel || lateral) && sql->session->status != -ERR_AMBIGUOUS) { > @@ -4926,6 +4926,7 @@ rel_query(mvc *sql, sql_rel *rel, symbol > fnd = table_ref(sql, res, n->data.sym, lateral); > if (fnd && is_project(fnd->op)) > fnd->exps = list_merge(fnd->exps, pre_exps, (fdup)NULL); > + lateral_used = 1; > } else { > fnd = table_ref(sql, rel, n->data.sym, 0); > } > @@ -4934,7 +4935,7 @@ rel_query(mvc *sql, sql_rel *rel, symbol > > if (!fnd) > break; > - if (res && !lateral) > + if (res && !lateral_used) > res = rel_crossproduct(sql->sa, res, fnd, op_join); > else > res = fnd; > _______________________________________________ > checkin-list mailing list > checkin-list(a)monetdb.org > https://www.monetdb.org/mailman/listinfo/checkin-list >

1 0

Announcement: New Dec2016-SP4 Bugfix release of MonetDB suite
by Sjoerd Mullender 01 May '17

01 May '17

The MonetDB team at CWI/MonetDB BV is pleased to announce the Dec2016-SP4 bugfix release of the MonetDB suite of programs. More information about MonetDB can be found on our website at <http://www.monetdb.org/>. For details on this release, please see the release notes at <http://www.monetdb.org/Downloads/ReleaseNotes>. As usual, the download location is <http://dev.monetdb.org/downloads/>. Dec 2016-SP4 bugfix release MonetDB5 Server * Fixed a bug causing a crash during cleanup when mserver5 is stopped with monetdb stop database. * Some memory leaks were plugged. MonetDB Common * A potential deadlock was fixed in order index creation. * A bug that could happen during recovery of the write-ahead log (WAL) was fixed. See changeset 98ad79c555cc for details. SQL * Some memory leaks were plugged. * Upgrade code was added for an old change in the sys.settimeout function. * A bug was fixed with the automatic "vacuum" operation on system tables. Bug Fixes * 6110: cast of a SQL boolean value to a string or clob or (var)char is wrong * 6254: Crash (and assertion failure) after querying a view which uses a correlated subquery in the select-list * 6256: Assertion Trigger on FULL OUTER JOIN with more than two BETWEEN clauses * 6257: wrong count values (1 instead of 0) for correlated aggregation queries * 6258: Vulnerability in FITS and NETCDF data vaults * 6259: crash on select query from sqlitelogictests * 6260: Sqlitelogictest crash * 6288: Function cannot find column in merge table * 6295: msqldump writes unescaped timestamp values when using inserts

1 0

Re: MonetDB: viewless - remove BATslice of candidate lists from gdk_...
by Roberto Cornacchia 28 Apr '17

28 Apr '17

No idea what viewless branch is aiming at, but my 5 1/2th sense tells me there is something interesting here.. ;) On 27 Apr 2017 5:56 pm, "lsidir" <commits+lsidir=gmail.com(a)monetdb.org> wrote: > Changeset: 63c8d8dbed74 for MonetDB > URL: http://dev.monetdb.org/hg/MonetDB?cmd=changeset;node=63c8d8dbed74 > Modified Files: > gdk/gdk_firstn.c > Branch: viewless > Log Message: > > remove BATslice of candidate lists from gdk_firstn > > > diffs (45 lines): > > diff --git a/gdk/gdk_firstn.c b/gdk/gdk_firstn.c > --- a/gdk/gdk_firstn.c > +++ b/gdk/gdk_firstn.c > @@ -140,7 +140,7 @@ BATfirstn_unique(BAT *b, BAT *s, BUN n, > if (n >= (BUN) (candend - cand)) { > /* trivial: return the candidate list (the > * part that refers to b, that is) */ > - return BATslice(s, > + return CANDslice(s, > (BUN) (cand - (const oid *) > Tloc(s, 0)), > (BUN) (candend - (const oid *) > Tloc(s, 0))); > } > @@ -163,12 +163,12 @@ BATfirstn_unique(BAT *b, BAT *s, BUN n, > /* return copy of first relevant part > * of candidate list */ > i = (BUN) (cand - (const oid *) Tloc(s, > 0)); > - return BATslice(s, i, i + n); > + return CANDslice(s, i, i + n); > } > /* return copy of last relevant part of > * candidate list */ > i = (BUN) (candend - (const oid *) Tloc(s, 0)); > - return BATslice(s, i - n, i); > + return CANDslice(s, i - n, i); > } > bn = COLnew(0, TYPE_void, n, TRANSIENT); > if (bn == NULL) > @@ -374,7 +374,7 @@ BATfirstn_unique_with_groups(BAT *b, BAT > if (BATtdense(g)) { > /* trivial: g determines ordering, return initial > * slice of s */ > - return BATslice(s, 0, n); > + return CANDslice(s, 0, n); > } > > CANDINIT(b, s, start, end, cnt, cand, candend); > @@ -916,7 +916,7 @@ BATfirstn_grouped_with_groups(BAT **topn > if (BATtdense(g)) { > /* trivial: g determines ordering, return initial > * slice of s */ > - bn = BATslice(s, 0, n); > + bn = CANDslice(s, 0, n); > gn = gids ? BATslice(g, 0, n) : NULL; > if (bn == NULL || (gids != NULL && gn == NULL)) { > BBPreclaim(bn); > _______________________________________________ > checkin-list mailing list > checkin-list(a)monetdb.org > https://www.monetdb.org/mailman/listinfo/checkin-list >

2 2

New Jul2017 stable branch created
by Sjoerd Mullender 24 Apr '17

24 Apr '17

I have created a new stable branch for the next feature release. This branch was created in order to stabilize the code before the feature release. From now on *only* bug fixes may go to this new stable branch. I reserve the right to undo any changes that do not comply with this rule. I also reserve the right to admit changes that are not strictly bug fixes. The name for the branch is Jul2017. So in each of your stable checkout directories you can do hg pull hg update -rJul2017 to update to this new branch. -- Sjoerd Mullender

3 2

MonetDB Intra-Query Checkpointing
by Playfair, Daniel 13 Apr '17

13 Apr '17

Hi all & Stefan, I mentioned my intra-query checkpointing work in my query about database sizing, Stefan expressed interest so I thought I'd give a brief overview in a separate topic for anyone interested... This work was inspired by: - the observation that current database HA doesn't do much to protect in-flight queries, and - the existence of databases such as MonetDB, and SAP HANA that materialize their results and have graph-like query plans. My work aims to leverage potential lower cost of checkpointing these materialized intermediates (since there is no materialization required), as well as the graph like structure of query plans to provide HA for queries. This is intended to benefit long running queries, since the cost to simply repeating short queries is low. The first aspect of my work is algorithms to produce checkpoint plans from query plan graphs. Currently this planning is an offline step, based on the output of query profiling. This work so far has discovered some interesting counter-intuitive but useful placement of checkpoints. The second aspect is my implementation for evaluating the general mechanism, and specific algorithms. I have implemented this in MonetDB. Some details: - I produce the graph structure for my algorithms from the exported MAL for a query. - Clients connect through a load balancer ("Persephone") that I have written for MonetDB. - This load balancer also orchestrates the mechanism it: o injects checkpoint plans into the database o receives checkpoints o detects failures o initiates replays on another database node by sending it a replay plan (checkpoints + skipped operations) - This process is transparent to clients and they require no modifications. - My modifications to MonetDB: (which shift most of the decision making to Persephone) o add a side-channel connection facilitating this orchestration. o Modify the mal interpreter: ? When a query matches a checkpoint plan, the selected intermediates will be transmitted using the side-channel connection to Persephone. ? To follow replay plans: using previously checkpointed intermediates, and entirely skipping operations that are now redundant due to checkpoints Hopefully this answers some of your questions Stefan. I'm happy to answer questions if anyone is interested. Thanks, ~ Daniel.

1 0

Sizing benchmarks
by Playfair, Daniel 13 Apr '17

13 Apr '17

Hi, I was wondering is there any guidance or rules of thumb for setting the parameters (database size, number of simultaneous users) of benchmarks in MonetDB in relation to system properties (RAM, CPU, Disk, Network)? Specifically I'm looking to run microbenchmarks with TPC-H, looking at performance of each query in isolation, so for my setup a RAM vs database size relationship is probably most important. (This is to evaluate the impact of intra-query checkpointing capabilities I've added to a modified version of MonetDB as part of my PhD work.) Thanks, ~ Daniel.

2 2

Announcement: New Dec2016-SP3 Bugfix release of MonetDB suite
by Sjoerd Mullender 31 Mar '17

31 Mar '17

The MonetDB team at CWI/MonetDB BV is pleased to announce the Dec2016-SP3 bugfix release of the MonetDB suite of programs. More information about MonetDB can be found on our website at <http://www.monetdb.org/>. For details on this release, please see the release notes at <http://www.monetdb.org/Downloads/ReleaseNotes>. As usual, the download location is <http://dev.monetdb.org/downloads/>. Dec 2016-SP3 bugfix release MonetDB Common * Fixed a bug when appending string bats that are fully duplicate eliminated. It could happend that the to-be-appended bat had an empty string at an offset and at that same offset in the to-be-appended-to bat there happened to be a (sequence of) NULL(s). Then this offset would be used, even though it might nog be the right offset for the empty string in the to-be-appended-to bat. This would result in multiple offsets for the empty string, breaking the promise of being duplicate eliminated. Bug Fixes * 6138: Weak duplicate elimination in string heaps > 64KB * 6183: ResultSet returns double quoted column name if name contains space characters * 6216: Assertion raised (sqlsmith) * 6219: Crash in rel_optimizer (sqlsmith) * 6227: Monetdb fails on remote tables * 6228: mclient crashes if real column is multiplied by it itself * 6229: ANALYZE, unexpected end of input * 6230: ANALYZE, syntax error * 6237: semijoin with empty right bat does not return immediately * 6242: Crash on rel_reduce_groupby_exps (sqlsmith) * 6243: Static optimization gives wrong result (1 + NULL = -127) * 6245: Nested query crashes all versions of MonetDB or gives wrong result starting from Dec2016-SP2 * 6246: update statements: references to a table do not bind to its alias * 6247: Type analysis issue (sqlsmith) * 6248: update statements: the semantic stage does not resolve the relation in the from clause * 6250: Assertion failure when querying a Blob column with order by DESC * 6251: Crash after adding an ordered index on sys.statistics column and querying sys.statistics * 6253: FITS Data Vaults does not work when using user/pw and other than sys schema name

1 0

BBP.dir file in MonetDB
by Senthil Vidhiyakar 31 Mar '17

31 Mar '17

Can someone tell me the need and file structure of BBP.dir file and the kind of information it contains.? Thank you, Senthil Vidhiyakar. S

2 1

File system in MonetDB
by Senthil Vidhiyakar 30 Mar '17

30 Mar '17

I want to know the file system or how the data is stored in MonetDB. I know data for each column is stored in separate files. But I want to know what those .*thead *and .*tail *files inside BAT folder of the monetDB data folder mean and how data is stored in those files. Please help me ASAP. Thank you.

2 1

Columnar Storage layout of MonetDB
by Senthil Vidhiyakar 21 Mar '17

21 Mar '17

I want to know the file layout of columnar data storage in MonetDB. I know each column is stored in separate files. Actually I want to know the exact layout as in a block diagram to get a clear picture of the storage layout. Helps are appreciated. Regards, Senthil Vidhiyakar. S

4 5

Announcement: New Dec2016-SP2 Bugfix release of MonetDB suite
by Sjoerd Mullender 03 Mar '17

03 Mar '17

The MonetDB team at CWI/MonetDB BV is pleased to announce the Dec2016-SP2 bugfix release of the MonetDB suite of programs. More information about MonetDB can be found on our website at <http://www.monetdb.org/>. For details on this release, please see the release notes at <http://www.monetdb.org/Downloads/ReleaseNotes>. As usual, the download location is <http://dev.monetdb.org/downloads/>. Dec 2016-SP2 bugfix release Bug Fixes * 4034: argnames array in rapi.c has fixed length (that was too short) * 6080: mserver5: rel_bin.c:2391: rel2bin_project: Assertion `0' failed. * 6081: Segmentation fault (core dumped) * 6082: group.subgroup is undefined if group by is used on an expression involving only constants * 6111: Maximum number of digits for hge decimal is listed as 39 in sys.types. Should be 38 as DECIMAL(39) is not supported. * 6112: Crash upgrading Jul2015->Jun2016 * 6130: Query rewriter crashes on a NULL pointer when having a correlated subquery * 6133: A crash occurs when preparing an INSERT on joined tables during the query semantic phase * 6141: Getting an error message regarding a non-GROUP-BY column rather than an unknown identifier * 6177: Server crashes * 6186: Null casting causes no results (silent server crash?) * 6189: Removing a NOT NULL constraint from a PKey column should NOT be allowed * 6190: CASE query crashes database * 6191: MT_msync failed with "Cannot allocate memory" * 6192: Numeric column stores wrong values after adding large numbers * 6193: converting to a smaller precision (fewer or no decimals after decimal point) should round/truncate consistently * 6194: splitpart returns truncated last part if it contains non ascii caracters * 6195: Cast from huge decimal type to smaller returns wrong results * 6196: Database crashes after generate_series query * 6198: COALESCE could be more optimized * 6201: MonetDB completely giving up on certain queries - no error and no result * 6202: querying a table with an ordered index on string/varchar column crashes server and makes server unrestartable! * 6203: copy into: Failed to import table Leftover data 'False' * 6205: Integer addition overflow * 6206: casting strings with more than one trailing zero ('0') to decimal goes wrong * 6209: Aggregation over complex OR expressions produce wrong results * 6210: Upgrading a database from Jun2015 or older crashes the server * 6213: SQLsmith causes server to crash * 6217: Segfault in rel_optimizer (sqlsmith) * 6218: grouped quantiles with all null group causes following groups to return null * 6224: mal_parser: cannot refer to types containing an underscore

1 0

WARNING for users of the default branch
by Sjoerd Mullender 02 Mar '17

02 Mar '17

Databases created with the default branch before changeset f054b7310be7 (Mar 1, 2017) are *not* compatible with the default branch from that changeset on. The changeset is a merge of fixes from the Dec2016 branch, and one of the fixes introduced a version number update of the BBP.dir file. This update was incompatible with an earlier update of the version number that was exclusive to the default branch. There are two possible solutions: - dump your database(s) with a version from before the chnangeset, and restore with a version after. - manually update the version number in the BBP.dir file and ignore the possibility that the bug that was fixed may be in your database. Manually fixing the BBP.dir file involves incrementing the number on the first line of the file from 25118 to 25119. Needless to say, this is not recommended. Remember, there are no guarantees when using the default branch. In fact, there are also no guarantees when using any other branch. The only thing we promise is that we will do our best to make sure the actual releases work and can be upgraded properly. -- Sjoerd Mullender

1 0

monetdb.org system maintenance
by Sjoerd Mullender 01 Mar '17

01 Mar '17

Tomorrow, March 2, we will be upgrading the hardware for the system on which all monetdb.org infrastructure runs. Expect some downtime. -- Sjoerd Mullender

1 0

Announcement: New Dec2016-SP1 Bugfix release of MonetDB suite
by Panagiotis Koutsourakis 26 Jan '17

26 Jan '17

The MonetDB team at CWI/MonetDB BV is pleased to announce the Dec2016-SP1 bugfix release of the MonetDB suite of programs. More information about MonetDB can be found on our website at <http://www.monetdb.org/>. For details on this release, please see the release notes at <http://www.monetdb.org/Downloads/ReleaseNotes>. As usual, the download location is <http://dev.monetdb.org/downloads/>. Dec 2016-SP1 bugfix release Bug Fixes * 4039: Slow mserver5 start after drop of tables (> 1 hour) * 4048: Segfault on vacuum with parallel updates * 6079: pushselect optimizer bug on MAL snippet * 6140: INNER JOIN gives the results of a CROSS JOIN * 6150: Query giving wrong results, extra records are appearing * 6175: The program can't start because python27.dll is missing from your computer. * 6178: AVG + GROUP BY returns NULL for some records that should have results * 6179: mergetable optimizer messes up sample * 6182: sys.shutdown triggers assertion in clients.c * 6184: Incorrect result set - Extra records in result set

1 0

python3 client bug+fix
by Bas Bloemsaat 23 Dec '16

23 Dec '16

Hi, Working with MonetDB I encountered a problem with queries containing utf-8 characters. Some queries errored. I traced the error back to mapi.py module for python3. The problem was that encode() was applied after splicing the packet. With encode, any utf-8 characters are expanded to multiple octets. For a given block that exceeded MAX_PACKAGE_LENGTH this resulted in the last few octets falling off a package. The patch below remedies this, please apply. Regards, Bas Bloemsaat --- /srv/src/monetdb/MonetDB-11.23.13/clients/python3/monetdb/mapi.py 2016-10-07 11:00:50.000000000 +0200 +++ /home/bas/.pyenv/versions/ent_venv1/lib/python3.5/site-packages/monetdb/mapi.py 2016-12-23 12:02:53.431357999 +0100 @@ -275,8 +275,9 @@ class Connection(object): def _putblock_inet(self, block): pos = 0 last = 0 + block = block.encode() while not last: - data = block[pos:pos + MAX_PACKAGE_LENGTH].encode() + data = block[pos:pos + MAX_PACKAGE_LENGTH] length = len(data) if length < MAX_PACKAGE_LENGTH: last = 1

2 2

Announcement: New Dec2016 Feature release of MonetDB suite
by Panagiotis Koutsourakis 21 Dec '16

21 Dec '16

The MonetDB team at CWI/MonetDB BV is pleased to announce the Dec2016 feature release of the MonetDB suite of programs. More information about MonetDB can be found on our website at <http://www.monetdb.org/>. For details on this release, please see the release notes at <http://www.monetdb.org/Downloads/ReleaseNotes>. As usual, the download location is <http://dev.monetdb.org/downloads/>. Dec 2016 feature release MonetDB5 Server * Removed the zorder module with functions zorder.encode, zorder.decode_x and zorder.decode_y. * Removed command bbp.getHeat(). * Removed bat.setColumn with two arguments and bat.setRole. Use bat.setColumn with one argument instead. * Removed function BKCappend_reverse_val_wrap: it was unused. * The "wrd" type has been removed from GDK and MAL. The type was defined to be a 32 bit integer on 32 bit architectures and a 64 bit integer on 64 bit architectures. We now generally use "lng" (always 64 bits) where "wrd" was used. * Keep a collection of full traces. Each time the SQL user applies the TRACE option, the full json trace is retained within the <dbpath>/<dbname>/sql_traces Build Environment * New packages MonetDB-python2 (Fedora) and monetdb-python2 (Debian/Ubuntu) have been created for Python 2 integration into MonetDB. * With OID size equal to ABI/word size, mserver5 does not need to print the OID size, anymore. * Removed obsolete code associated with long gone static linking option. * Removed configure option --enable-oid32 to compile with 32 bit OIDs on a 64 bit architecture. . * The Perl, PHP, and Python clients, and the JDBC driver each now have their own repositories and release cycles. The Python client is maintained by Gijs Molenaar on Github (https://github.com/gijzelaerr/pymonetdb) the other clients are maintained by CWI/MonetDB on our own server (https://dev.monetdb.org/hg/monetdb-java, https://dev.monetdb.org/hg/monetdb-perl, https://dev.monetdb.org/hg/monetdb-php) MonetDB Common * The tnokey values must now be 0 if it is not known whether all values in a column are distinct. * The 2-bit tkey field in the bat descriptor has been split into two single bit fields: tkey and tunique. The old tkey&BOUND2BTRUE value is now stored in tunique. * Implemented conversion to str from any type (not just the internal types). * VALcopy and VALinit both return their first argument on success or (and that's new) NULL on (allocation) failure. * BATattach now can also create a str BAT from a file consisting of null-terminated strings. The input file must be encoded using UTF-8. * BATattach now copies the input file instead of "stealing" it. * Removed the lastused "timestamp" from the BBP. * Removed batStamp field from BAT descriptor, and removed the BBPcurstamp function. * Removed unused functions BBPhot and BBPcold. * Removed BATderiveTailProps and BATderiveProps. Just set the properties you know about, or use BATsettrivprop. * Removed the macro BUNfirst. It can be replaced by 0. * Changed BATroles by removing the argument to set the name of the head column. * The head column is now completely gone. MonetDB is completely "headless". * The format of the BBP.dir file was simplified. Since the head column is VOID, the only value that needs to be stored is the head seqbase. * BATs now only have a single (logical) name. * The function BATmirror is gone. The HEAD column is always VOID (with a non-nil seqbase) and the TAIL column carries the data. All functions that deal with data work on the TAIL column. * BATkey now works on the TAIL column instead of the HEAD column. * Replaced BATseqbase with BAThseqbase and BATtseqbase, the former for setting the seqbase on the HEAD, the latter for setting the seqbase on the TAIL. * Replaced function BATnew with COLnew with slightly different arguments: the first argument of COLnew is the SEQBASE of the head column (which is always VOID). * The "wrd" type has been removed from GDK and MAL. The type was defined to be a 32 bit integer on 32 bit architectures and a 64 bit integer on 64 bit architectures. We now generally use "lng" (always 64 bits) where "wrd" was used. SQL * Removed functions sys.zorder_encode, sys.zorder_decode_x, and sys.zorder_decode_y. * The experimental recycler code is moved to the attic. * The syntax of bat.new(:oid,:any) has been changed by dropping the superflous :oid. All BATs are now binary associations between a void column and a materialized value column. (except for the internal :bat[:void,:void] representation of simple oid ranged tails.) Bug Fixes * 3357: Implement setQueryTimeout() * 3445: Add support for database name to dotmonetdb file * 3597: SQL to MAL listing looses types * 3973: JDBC hangs * 3976: Performance enhancement to LIKE without wildcards * 4005: Correlated update causes incorrect null constraint violation * 4016: merge table only optimises for point query * 4040: sys.storage call can take a long time * 4047: Segfault when updating a dropped table * 4050: Database corruption when running low on inode * 4057: missing bulk operations between constant and bat * 4061: SIGSEGV in candscan_lng * 4066: Deadlocked monetdbd * 6068: Error message about incompatible BBP version should be clearer * 6069: query with union all silently crashes * 6070: setting negative session query timeout should not be possible/allowed * 6071: where clause with cast and floor fails to sigsegv * 6072: Bind to UPD delta column does not get/show type information in EXPLAIN * 6073: Missing type information for constants in MAL explain * 6074: SET ROLE command does not work * 6075: gdk_calc.c:13113: BATcalcifthenelse_intern: Assertion `col2 != NULL' failed. * 6076: rel_optimizer.c:5426: rel_push_project_up: Assertion `e' failed. * 6077: mserver5: rel_optimizer.c:5444: rel_push_project_up: Assertion `e' failed. * 6078: rel_bin.c:2402: rel2bin_project: Assertion `0' failed. * 6084: Merge table point to wrong columns if columns in partition tables are deleted * 6108: monetdb5-sql sysv init script does not wait for shutdown * 6114: segfault raised in the query rewriter due to a null pointer * 6115: Assertion hit in the codegen * 6116: Codegen does not support certain kind of selects on scalar subqueries * 6117: Assertion hit in the query rewriting stage during the push down phase * 6118: SIGSEGV in strPut due to shared heap * 6119: Assertion hit in the MAL optimiser on a complex query * 6120: QUANTILE() treats NULL as if it is zero * 6121: SELECT a.col IN ( b.col FROM b ) FROM a statements with no error but no result * 6123: Assertion hit in the codegen #2 * 6124: CASE <column> WHEN NULL THEN 0 ELSE 1 END returns wrong result * 6125: Stack overflow in the query rewriter with a query having an OR condition and a nested SELECT subexpression * 6126: batcalc.== can't handle void BATs * 6139: Debian libmonetdb13 conflicts with libmonetdb5-server-geom

1 0

RE: Need Help on enabling GIS (MonetDB/GIS) module in MonetDB > (Joweria Nakibuuka) RE: Need Help on enabling GIS (MonetDB/GIS) module in MonetDB > (Joweria Nakibuuka)
by Joweria Nakibuuka 01 Dec '16

01 Dec '16

joweria@ubuntu:~$ mserver5 --dbpath=/tmp/MyDBfarm/MyDB </dev/null # MonetDB 5 server v11.24.0 # This is an unreleased version # Serving database 'MyDB', using 2 threads # Compiled for x86_64-unknown-linux-gnu/64bit with 128bit integers # Found 1.470 GiB available main-memory. # Copyright (c) 1993-July 2008 CWI. # Copyright (c) August 2008-2016 MonetDB B.V., all rights reserved # Visit http://www.monetdb.org/ for further information # Listening for connection requests on mapi:monetdb://127.0.0.1:50000/ # MonetDB/GIS module loaded # MonetDB/SQL module loaded >joweria@ubuntu:~$ joweria@ubuntu:~$ mserver5 --dbpath=/tmp/MyDBfarm/MyDB </dev/null # MonetDB 5 server v11.24.0 # This is an unreleased version # Serving database 'MyDB', using 2 threads # Compiled for x86_64-unknown-linux-gnu/64bit with 128bit integers # Found 1.470 GiB available main-memory. # Copyright (c) 1993-July 2008 CWI. # Copyright (c) August 2008-2016 MonetDB B.V., all rights reserved # Visit http://www.monetdb.org/ for further information # Listening for connection requests on mapi:monetdb://127.0.0.1:50000/ # MonetDB/GIS module loaded # MonetDB/SQL module loaded But when i try to reach another database as shown below, the module is not loaded joweria@ubuntu:~$ mserver5 --dbpath=/home/joweria/Spartial/mydbfarm/Gisbase !FATAL: GDKlockHome: Database lock '.gdk_lock' denied joweria@ubuntu:~$ sudo su [sudo] password for joweria: root@ubuntu:/home/joweria# mserver5 --dbpath=/home/joweria/ Spartial/mydbfarm/Gisbase # MonetDB 5 server v11.24.0 # This is an unreleased version # Serving database 'Gisbase', using 2 threads # Compiled for x86_64-unknown-linux-gnu/64bit with 128bit integers # Found 1.470 GiB available main-memory. # Copyright (c) 1993-July 2008 CWI. # Copyright (c) August 2008-2016 MonetDB B.V., all rights reserved # Visit http://www.monetdb.org/ for further information # Listening for connection requests on mapi:monetdb://127.0.0.1:50000/ # MonetDB/SQL module loaded > Also when i run with my user priviledges

1 0

Re: developers-list Digest, Vol 52, Issue 1
by Joweria Nakibuuka 01 Dec '16

01 Dec '16

joweria@ubuntu:~$ mserver5 --dbpath=/tmp/MyDBfarm/MyDB # MonetDB 5 server v11.24.0 # This is an unreleased version # Serving database 'MyDB', using 2 threads # Compiled for x86_64-unknown-linux-gnu/64bit with 128bit integers # Found 1.470 GiB available main-memory. # Copyright (c) 1993-July 2008 CWI. # Copyright (c) August 2008-2016 MonetDB B.V., all rights reserved # Visit http://www.monetdb.org/ for further information # Listening for connection requests on mapi:monetdb://127.0.0.1:50000/ # MonetDB/GIS module loaded # MonetDB/SQL module loaded 2016-12-01 13:00 GMT+02:00 <developers-list-request(a)monetdb.org>: > Send developers-list mailing list submissions to > developers-list(a)monetdb.org > > To subscribe or unsubscribe via the World Wide Web, visit > https://www.monetdb.org/mailman/listinfo/developers-list > or, via email, send a message with subject or body 'help' to > developers-list-request(a)monetdb.org > > You can reach the person managing the list at > developers-list-owner(a)monetdb.org > > When replying, please edit your Subject line so it is more specific > than "Re: Contents of developers-list digest..." > > > Today's Topics: > > 1. RE: Need Help on enabling GIS (MonetDB/GIS) module in MonetDB > (Joweria Nakibuuka) > 2. Re: Need Help on enabling GIS (MonetDB/GIS) module in MonetDB > (Stefan Manegold) > > > ---------------------------------------------------------------------- > > Message: 1 > Date: Thu, 1 Dec 2016 12:11:49 +0200 > From: Joweria Nakibuuka <joweria.nakibuuka(a)unitn.it> > To: developers-list(a)monetdb.org > Subject: RE: Need Help on enabling GIS (MonetDB/GIS) module in MonetDB > Message-ID: > <CANbNUqbw+N3=tb929XArQ8q+AiCG-o5DYNR9Ndk2POChHpdi1g@ > mail.gmail.com> > Content-Type: text/plain; charset="utf-8" > > Dear Admin, > I am running Ubuntu linux 14.04 LTS, x86_64-Ubuntu-linux-gnu/64bit with > 128bit integers. i recently installed MonetDB5 default branch from the > mercurial repository. > > I tried to rebuild and enable geom module but to no avail, it shows no > error, after compilation it shows that the module is enabled,However when i > run the mserver the monetDB/GIS module is not loaded as expected. I am > stuck and i don't seem to know how to proceed forward in order to run my > spatial queries > > Are there any setting i need to do explicitly to load this module? > > the command i used: ./configure --prefix=/home/joweria/MonetDB-Nov2016-SP4 > --enable-geom=yes --enable-pyintegration=yes --with-geos=yes > > and then after compilation under the LDFLAGS-enabled componets > geom is one of the enabled componet but when i run the mserver5 the module > is not loaded as expected. > > I rebuild monetDB again and for first time i run the mserver5 commans > and the GIS module was successfully loaded ie (MonetDB/SQL/GIS module is > loaded) but when i closed the session (\q) and ran the same command again > the module was not loaded. i tried exporting paths to no avail. what could > be the problem? please i need your kind help. > > Thank you > > Regards, > Joweria nakibuuka >

2 2

RE: Need Help on enabling GIS (MonetDB/GIS) module in MonetDB
by Joweria Nakibuuka 01 Dec '16

01 Dec '16

Dear Admin, I am running Ubuntu linux 14.04 LTS, x86_64-Ubuntu-linux-gnu/64bit with 128bit integers. i recently installed MonetDB5 default branch from the mercurial repository. I tried to rebuild and enable geom module but to no avail, it shows no error, after compilation it shows that the module is enabled,However when i run the mserver the monetDB/GIS module is not loaded as expected. I am stuck and i don't seem to know how to proceed forward in order to run my spatial queries Are there any setting i need to do explicitly to load this module? the command i used: ./configure --prefix=/home/joweria/MonetDB-Nov2016-SP4 --enable-geom=yes --enable-pyintegration=yes --with-geos=yes and then after compilation under the LDFLAGS-enabled componets geom is one of the enabled componet but when i run the mserver5 the module is not loaded as expected. I rebuild monetDB again and for first time i run the mserver5 commans and the GIS module was successfully loaded ie (MonetDB/SQL/GIS module is loaded) but when i closed the session (\q) and ran the same command again the module was not loaded. i tried exporting paths to no avail. what could be the problem? please i need your kind help. Thank you Regards, Joweria nakibuuka

2 1

MAL: How to interate through group values?
by Gatis Ozoliņš 30 Nov '16

30 Nov '16

I am trying to create my own MAL aggregate function. I have implemented main function and it works OK: function group_concat(b:bat[:oid, :str],s:bat[:oid, :str]):str; sep := ""; barrier (h, t) := iterator.new(s); sep := t; exit (h, t); value := nil:str; barrier (h, t) := iterator.new(b); barrier bnormal := value; value := value + sep; value := value + t; exit bnormal; barrier bnil := calc.isnil(value); value := t; exit bnil; redo (h, t) := iterator.next(b); exit (h, t); return value; end group_concat; Now I need to implemented subfunction with following definition: function subgroup_concat(b:bat[:oid,:str],s:bat[:oid,:str],g:bat[:oid,:oid],e:bat[:oid,:any_2],skip_nils:bit) :bat[:oid,:str]; I can iterate over all groups g, but I can't find correct MAL function to get all values (b,s) for this specific group. Which MAL function would return list of oid for specific group g entry ? Gatis

1 0

.gdk_lock and the GDK locking scheme
by Eyal Rozenberg 28 Nov '16

28 Nov '16

I used to think .gdk_lock was used by mserver the same way Linux daemons use lock files - the filesystem enforces serialization, so if you do: 1. if (lock file present) then locking failed 2. create the lock file 3. if (creation failed) then locking failed 4. locking successful that's a valid non-blocking locking procedure. However, it appears that: * .gdk_lock is not always empty, and can have text such as USR=12096163 PID=5844 TIME=Thu Nov 24 15:35:08 2016 @ LOGON USR=12096163 PID=5844 TIME=Thu Nov 24 15:42:26 2016 @ LOGOFF * If .gdk_lock is not always created on DB startup, nor deleted on shutdown. * The locking using .gdk_lock is actually an flock() on its 5th byte (offset 4) and never any other one. * The GDKunlockhome() file will cause a second file descriptor opening in addition to GDKlockfile So I'm a bit confused about the locking scheme. Help? Eyal

1 0

Questions about BATs and their names in BBP's
by Eyal Rozenberg 24 Nov '16

24 Nov '16

So, I'm listing the BATs in a BBP (of a TPCH database), and I'm noticing several strange things: 1. There's a h-u-g-e number of BATs. Actually I knew this to be the case already, since BBPs have so many files, but, it's still striking. 863 BATs for a database with 61 bona fide columns (in lineitem, customer etc.) 2. There are a lot of BATs with null storage. And I do mean a lot - 514 out of 863. What are these? why do they need to exist? And note this is a database in which nothing has occurred except one-time loading. 3. Most BATs have dummy generated names, e.g. tmp_547 - not, say, a concatenation of their table name and column name. Yet - some BATs _do_ have meaningful names: environment_key, sql_dsnapshots, stat_opt_calls and so on. Why not have meaningful names for all BATs then? and lastly, and perhaps most importantly: 5. What's the best/most appropriate/most quick-and-dirty way of obtaining table and column names for those BATs which have them? 6. If it's through the storage table - is there a way to determine which BATs correspond to its columns (table, column, location at the very least)?

2 1

Can I expect to be able to use BBP code without GDKinit'ing?
by Eyal Rozenberg 24 Nov '16

24 Nov '16

Bottom line question: BBPinit() is not exposed in gdk_bbp.h, but rather only called from GDKinit() (which sees it through gdk_private.h). I just want to load data from persisted BAT files - not into MonetDB and not within the mserver5 process - so, speaking conceptually, I do not want to initialize the GDK, but do want to initialize the BBP. Questions: * Should BBPinit() work outside the scope of a GDKinit()? * If not, is it adaptable so as to allow this? * More generally, how much of GDK do I need to have running, just so as load persisted data into memory, using the code in gdk_bbp (or a slight variation thereof)? Now for the introduction and the motivation: As I mentioned a few MADADMs ago, I'll need to load persisted MonetDB columns into my GPU kernel testbench. This is relatively simple for numeric columns, given some scripting work to extract catalog data in a parsable format or to build named symlinks to columns (eg /path/to/dbfarm/tpch-sf-1/named_bats/lineitem/l_shipdate -> /path/to/dbfarm/tpch-sf-1/bat/12/34.tail). But it won't do for other kinds of data, most importantly strings - which I do want to work on. Plus, writing a wider-scope loader will let me avoid depending on MonetDB running for access to persisted columns. So, I'm writing a (selective) loader of persisted MonetDB columns, or a BBP loader if you will. My strategy is the following: 1. Copying the files in the GDK codebase which are necessary for building a binary which can call all code in gdk_bbp.h and not have unmet dependencies (this is mostly done). 2. Get that leaner slice of code, with a small main(), to actually work, i.e. not fail due to weird errors or result in junk data. 3. Peel away functions from the code I've copied from the repository which are not actually used. 4. Peel away the parts of the code which are not necessary for the actual loading - from within functions. In this initial stage this may involve work that becomes unnecessary when you have auxiliary data obtained from querying the DB (such as table-column name to BAT filename mapping). 5. Expand functionality and/or optimize performance after the peeling and/or C++ify for integration with my code 6. If other people / MDBS are interested, collaborate on making this support different BBP versions - so that the result is an inter-version export/forensics library.

2 1

Re: developers-list Digest, Vol 51, Issue 4
by Joweria Nakibuuka 23 Nov '16

23 Nov '16

the command i used: ./configure --prefix=/home/joweria/MonetDB-Nov2016-SP4 --enable-geom=yes --enable-pyintegration=yes --with-geos=yes and then after compilation under the LDFLAGS-enabled componets geom is one of the enabled componet 2016-11-10 13:00 GMT+02:00 <developers-list-request(a)monetdb.org>: > Send developers-list mailing list submissions to > developers-list(a)monetdb.org > > To subscribe or unsubscribe via the World Wide Web, visit > https://www.monetdb.org/mailman/listinfo/developers-list > or, via email, send a message with subject or body 'help' to > developers-list-request(a)monetdb.org > > You can reach the person managing the list at > developers-list-owner(a)monetdb.org > > When replying, please edit your Subject line so it is more specific > than "Re: Contents of developers-list digest..." > > > Today's Topics: > > 1. Need Help on enabling GIS (MonetDB/GIS) module in MonetDB > (Joweria Nakibuuka) > 2. Re: Need Help on enabling GIS (MonetDB/GIS) module in MonetDB > (Ying Zhang) > > > ---------------------------------------------------------------------- > > Message: 1 > Date: Wed, 9 Nov 2016 17:39:17 +0100 > From: Joweria Nakibuuka <joweria.nakibuuka(a)unitn.it> > To: developers-list(a)monetdb.org > Subject: Need Help on enabling GIS (MonetDB/GIS) module in MonetDB > Message-ID: > <CANbNUqZfu_pfzsZ=xUesfyLQPw3Ji3DW_fYeNOSZwfkTw= > 1Hsg(a)mail.gmail.com> > Content-Type: text/plain; charset="utf-8" > > Dear Admin, > I am running Ubuntu linux 14.04 LTS, x86_64-Ubuntu-linux-gnu/64bit with > 128bit integers. i recently installed MonetDB5 default branch from the > mercurial repository. > > I tried to rebuild and enable geom module but to no avail, it shows no > error, after compilation it shows that the module is enabled,However when i > run the mserver the monetDB/GIS module is not loaded as expected. I am > stuck and i don't seem to know how to proceed forward in order to run my > spatial queries > > Are there any setting i need to do explicitly to load this module? > > Thank you > > Regards, > Joweria nakibuuka >

2 2

Need Help on enabling GIS (MonetDB/GIS) module in MonetDB
by Joweria Nakibuuka 10 Nov '16

10 Nov '16

Dear Admin, I am running Ubuntu linux 14.04 LTS, x86_64-Ubuntu-linux-gnu/64bit with 128bit integers. i recently installed MonetDB5 default branch from the mercurial repository. I tried to rebuild and enable geom module but to no avail, it shows no error, after compilation it shows that the module is enabled,However when i run the mserver the monetDB/GIS module is not loaded as expected. I am stuck and i don't seem to know how to proceed forward in order to run my spatial queries Are there any setting i need to do explicitly to load this module? Thank you Regards, Joweria nakibuuka

2 1

Two new juicy assertions in the SQL front-end
by Dean De Leo 08 Nov '16

08 Nov '16

Dear MonetDB developers, I'm reporting two other assertions I hit in the front-end. The first is related to scalar subqueries, see query3.sql for a repro and backtrace3.txt. I guess it could be rather considered a feature request... BTW I appreciate this style of treating unsupported features, I'm starting to use in my changes as well :) The second occurs somewhere in the query rewriter when issuing the whole query, see query4.sql and backtrace4.txt. On Linux, tip of the default branch, debug build, empty tables, schema attached. Thanks, Dean

1 0

Two sweet crashes in the sql front-end
by Dean De Leo 08 Nov '16

08 Nov '16

Dear MonetDB developers, I would like to report a crash due to a null pointer using the attached schema.sql and query1a.sql, see backtrace1.txt. Eventually in the process of simplyfying the query to make this report, I stumbled upon a second assertion, see query2_crash_repro.sql and backtrace2.txt On Linux, tip of the default branch, debug build, empty tables. Dean

1 0

Re: MonetDB: malupgrade - Clear variables before resetting mb->vtop.
by Martin Kersten 07 Nov '16

07 Nov '16

WOW .... that took me quite some time nailing the example and looking into the wrong direction thx On 07/11/16 12:38, Sjoerd Mullender wrote: > Changeset: 8ac922886df2 for MonetDB > URL: http://dev.monetdb.org/hg/MonetDB?cmd=changeset;node=8ac922886df2 > Modified Files: > monetdb5/mal/mal_instruction.c > Branch: malupgrade > Log Message: > > Clear variables before resetting mb->vtop. > > > diffs (20 lines): > > diff --git a/monetdb5/mal/mal_instruction.c b/monetdb5/mal/mal_instruction.c > --- a/monetdb5/mal/mal_instruction.c > +++ b/monetdb5/mal/mal_instruction.c > @@ -197,13 +197,13 @@ freeMalBlk(MalBlkPtr mb) > mb->stmt[i] = NULL; > } > mb->stop = 0; > + for(i=0; i< mb->vtop; i++) > + if (isVarConstant(mb, i)) > + VALclear(&getVarConstant(mb,i)); > mb->vtop = 0; > mb->vid = 0; > GDKfree(mb->stmt); > mb->stmt = 0; > - for(i=0; i< mb->vtop; i++) > - if (isVarConstant(mb, i)) > - VALclear(&getVarConstant(mb,i)); > GDKfree(mb->var); > mb->var = 0; > > _______________________________________________ > checkin-list mailing list > checkin-list(a)monetdb.org > https://www.monetdb.org/mailman/listinfo/checkin-list >

1 0

Data lost
by Luciano Sasso 17 Oct '16

17 Oct '16

Hi, I am delete rows from table, with "delete from mytable where id=x", and rows deleted no return in select "select * from mytable where id=x", its OK ! Version "Jul2015-SP1" But, when restart database, deleted rows return in select "select * from mytable where id=x". Its bug ?, or i have execute vacuum command, how to execute vacuum command. ? -- Luciano Sasso Vieira Data Scientist & Solutions Architect luciano(a)gsgroup.com.br <http://www.gsgroup.com.br> | tel: 17 3353-0833 | 17 3202-4040 | cel: 17 99706-9335 www.gsgroup.com.br <http://www.gsgroup.com.br> --- Este email foi escaneado pelo Avast antivírus. https://www.avast.com/antivirus

2 1

New Dec2016 stable branch created
by Sjoerd Mullender 14 Oct '16

14 Oct '16

I have created a new stable branch for the next feature release. This branch was created in order to stabilize the code before the feature release. From now on *only* bug fixes may go to this new stable branch. I reserve the right to undo any changes that do not comply with this rule. I also reserve the right to admit changes that are not strictly bug fixes. The name for the branch is Dec2016. So in each of your stable checkout directories you can do hg pull hg update -rDec2016 to update to this new branch. -- Sjoerd Mullender

1 0

Announcement: New Jun2016-SP2 Bugfix release of MonetDB suite
by Sjoerd Mullender 13 Oct '16

13 Oct '16

The MonetDB team at CWI/MonetDB BV is pleased to announce the Jun2016-SP2 bugfix release of the MonetDB suite of programs. More information about MonetDB can be found on our website at <http://www.monetdb.org/>. For details on this release, please see the release notes at <http://www.monetdb.org/Downloads/ReleaseNotes>. As usual, the download location is <http://dev.monetdb.org/downloads/>. Jun 2016-SP2 bugfix release Build Environment * We now use the CommonCrypto library instead of the OpenSSL library on Darwin. Bug Fixes * 3939: Assert failure on concurrent queries when querying sys.queue * 4019: Casting a timestamp from a string results in NULL * 4025: expressions in the WHERE clause that evaluates incorrectly * 4038: After upgrade from 11.21.19, jdbc couldn't list tables for non sys users * 4044: Server crash when trying to delete a table has been added to a merge table with "cascade" at the end * 4049: Wrong results for queries with "OR" and "LEFT JOIN" * 4052: Infinite loop in rel_select * 4054: copy into file wrongly exports functions * 4058: Server crashes with a particular conditional query * 4059: Geom functions only visible by user monetdb * 4060: BAT leak in some aggregate queries * 4062: Error: SELECT: no such binary operator 1800 790 9186'like(varchar,varchar)' when used in query running in other schema than sys * 4064: Assertion: column not found * 4067: Relevant column name not printed when a CSV parsing error occurs * 4070: Extra condition in join predicate of explicit join produces wrong MAL code * 4074: Cannot use prepared statements when caching disabled * 6065: CTE with row number and union fails within MAL

1 0

Bulk load from memory
by Hermano ... 09 Oct '16

09 Oct '16

Hello I became very interested in MonetDB during my research, and I am considering developing my own MAL compiler for a system based on a data model my research group is working on in the future. For now, I've been trying to understand MAL and how the COPY INTO command works in MonetDB Initially, I wish to do a large bulk load from an array in memory directly into MonetDB. My first "closed box" approach was to do the following: -Create a binary integer file containing milions of integers in /dev/shm/ named teste.bin. It is a memory based device, so the data is expected to be in main memory. -Create a table in MonetDB: CREATE TABLE teste (val INT); -Try to do BINARY COPY FROM: COPY BINARY INTO teste FROM ('/dev/shm/teste.bin'). When I try to this I get the error: Failed to attach file /dev/shm/teste.bin BATattach: cannot rename heapfile !OS: Invalid cross-device link After that I understood that monetdb basically moves the entire file around, which can't be done with data from another device. I wish to know if there is any sort of functionality in MonetDB or even a MAL module that would allow me to read from some sort of shared memory buffer (kind vague I know) directly into a BAT file. The MAL program for the COPY command has an sql.importTable and sql.append calls, so I wonder if there is some variant of those that would allow me to load binary data from a memory buffer? Thanks in Advance

1 0

Fwd: lib_pyapi complains about undefined symbol PyInt_Type
by Dean De Leo 06 Oct '16

06 Oct '16

Hi, on my personal machine, when starting mserver5, monetdb complains with the following message: !SyntaxException:parseError: !LoaderException:loadLibrary:Loading error failed to open library pyapi (from within file '/linux/workspace/monetdb/build/lib/monetdb5/lib_pyapi.so'): /linux/workspace/monetdb/build/lib/monetdb5/lib_pyapi.so: undefined symbol: PyInt_Type !SyntaxException:parseError: !<identifier> expected I guess this is another case of conflict Python 2 vs 3. I am using ArchLinux where the default is v3. In my case a workaround seems to be forcing back python2 (see patch). Dean

2 1

GDK_mmap_minsize again
by Hannes Mühleisen 24 Sep '16

24 Sep '16

Hello list, we were wondering about the purpose of GDK_mmap_minsize when creating transient columns. The attached patch will always *try* to malloc/realloc a transient column but still fall back to memory-mapped files if malloc should fail. This dramatically improves performance. Any good reason why this should not be the default behaviour? Thanks, Mark and Hannes

5 5

eviction in Monetdb
by Narendra Pradeep 15 Sep '16

15 Sep '16

Hello , Need to know about monetdb eviction policy ? My understanding is Monetdb creates a buffer pool {BBP} and inserts all the bats that are used . But not sure when does Monetdb deletes the bat from the buffer pool and what eviction policy does it use (like LRU or based on some threshold) ? Thanks

2 1

Re: MonetDB: graph1 - Add the operator bat.getCount() to retrieve th...
by Stefan Manegold 14 Sep '16

14 Sep '16

Hi Dean, how is bat.getCount() different from aggr.count() ? ;-) Thanks! Stefan ----- On Sep 14, 2016, at 6:30 PM, Dean De Leo commits+dleo=cwi.nl(a)monetdb.org wrote: > Changeset: 9ac831deacb3 for MonetDB > URL: http://dev.monetdb.org/hg/MonetDB?cmd=changeset;node=9ac831deacb3 > Modified Files: > monetdb5/modules/kernel/bat5.c > monetdb5/modules/kernel/bat5.h > monetdb5/modules/kernel/bat5.mal > Branch: graph1 > Log Message: > > Add the operator bat.getCount() to retrieve the cardinality of a bat > > > diffs (48 lines): > > diff --git a/monetdb5/modules/kernel/bat5.c b/monetdb5/modules/kernel/bat5.c > --- a/monetdb5/modules/kernel/bat5.c > +++ b/monetdb5/modules/kernel/bat5.c > @@ -496,6 +496,19 @@ BKCgetColumnType(str *res, const bat *bi > } > > str > +BKCgetCount(lng *res, const bat *bid) > +{ > + *res = lng_nil; > + if (BBPcheck(*bid, "bat.getCount")) { > + BAT *b = BBPquickdesc(*bid, 0); > + > + if (b != NULL) > + *res = (lng) BATcount(b); > + } > + return MAL_SUCCEED; > +} > + > +str > BKCgetRole(str *res, const bat *bid) > { > BAT *b; > diff --git a/monetdb5/modules/kernel/bat5.h b/monetdb5/modules/kernel/bat5.h > --- a/monetdb5/modules/kernel/bat5.h > +++ b/monetdb5/modules/kernel/bat5.h > @@ -29,6 +29,7 @@ mal_export str BKCbun_inplace_force(bat > mal_export str BKCbat_inplace_force(bat *r, const bat *bid, const bat *rid, > const bat *uid, const bit *force); > mal_export str BKCgetCapacity(lng *res, const bat *bid); > mal_export str BKCgetColumnType(str *res, const bat *bid); > +mal_export str BKCgetCount(lng *res, const bat *bid); > mal_export str BKCgetRole(str *res, const bat *bid); > mal_export str BKCsetkey(bat *res, const bat *bid, const bit *param); > mal_export str BKCisSorted(bit *res, const bat *bid); > diff --git a/monetdb5/modules/kernel/bat5.mal b/monetdb5/modules/kernel/bat5.mal > --- a/monetdb5/modules/kernel/bat5.mal > +++ b/monetdb5/modules/kernel/bat5.mal > @@ -77,6 +77,10 @@ command getCapacity(b:bat[:any_1]):lng > address BKCgetCapacity > comment "Returns the current allocation size (in max number of elements) of a > BAT."; > > +command getCount(b:bat[:any_1]):lng > +address BKCgetCount > +comment "Returns the current tuple count of a BAT."; > + > command getColumnType( b:bat[:any_1] ) :str > address BKCgetColumnType > comment "Returns the type of the tail column of a BAT, as an integer type > number."; > _______________________________________________ > checkin-list mailing list > checkin-list(a)monetdb.org > https://www.monetdb.org/mailman/listinfo/checkin-list -- | Stefan.Manegold(a)CWI.nl | DB Architectures (DA) | | www.CWI.nl/~manegold/ | Science Park 123 (L321) | | +31 (0)20 592-4212 | 1098 XG Amsterdam (NL) |

1 0

BAT in memory
by Narendra Pradeep 30 Aug '16

30 Aug '16

Hello , I am reading monetdb source code . I need to know which function check whether the bat of the particular column lies in memory or not ? And how does it maps the bat with corresponding column ? Thanks . Regars , Naren

1 0

Monetdb printing values of variables in console or terminal
by Narendra Pradeep 23 Aug '16

23 Aug '16

Hai everyone , How to print messages in console or to write something in log file of the Monetdb . can someone help me with exact function ? I cant even write a word in logfile and console is there something with my build ? Regards , Pradeep

1 0

compression in monetdb
by Narendra Pradeep 11 Aug '16

11 Aug '16

Hai , Read that dictionary encoding occurs in monetdb . so 1) what is dictionary compression technique ? 2) Does it works for strings alone or other data structures too ? 3) I a mailing list it was stated that it works only till the size of the column in 128 MB for older release? Is this the same for current release ? 4) Does Monetdb uses any other compression technique ? 5) If yes where can I find the source of it (Precisely , which file in source ) Thanks in advance regards , Pradeep

1 0

Parameters in bind function
by Narendra Pradeep 08 Aug '16

08 Aug '16

Hello everyone , I am trying to understand the MAL statements for joining three tables . On which I came across bind function in MAL statement . I have a table lineitem with l_quantity in it . So what does the following statements means 1 ) ( X_94:bat[:oid,:oid] ,X_95:bat[:oid,:int] ) := sql.bind(X_4,"sys","lineitem","l_quantity",2,0,4); 2 ) (X_96:bat[:oid,:oid] ,X_97:bat[:oid,:int] ) := sql.bind(X_4,"sys","lineitem","l_quantity",2,1,4); what does the 5 th and 6 th argument means ? Regards , Pradeep

1 0

jdbc exception
by Panagiotis Koutsourakis 05 Aug '16

05 Aug '16

Hello, has anyone encountered something like the following exception before? java.lang.AssertionError: Server sent a block larger than BLOCKsize: 13456 > 8193 at nl.cwi.monetdb.mcl.net.MapiSocket$BlockInputStream.readBlock(MapiSocket.java:917) at nl.cwi.monetdb.mcl.net.MapiSocket$BlockInputStream.read(MapiSocket.java:966) at sun.nio.cs.StreamDecoder.readBytes(StreamDecoder.java:284) at sun.nio.cs.StreamDecoder.implRead(StreamDecoder.java:326) at sun.nio.cs.StreamDecoder.read(StreamDecoder.java:178) at java.io.InputStreamReader.read(InputStreamReader.java:184) at java.io.BufferedReader.fill(BufferedReader.java:161) at java.io.BufferedReader.readLine(BufferedReader.java:324) at java.io.BufferedReader.readLine(BufferedReader.java:389) at nl.cwi.monetdb.mcl.io.BufferedMCLReader.readLine(BufferedMCLReader.java:119) at nl.cwi.monetdb.mcl.net.MapiSocket.connect(MapiSocket.java:278) at nl.cwi.monetdb.mcl.net.MapiSocket.connect(MapiSocket.java:398) at nl.cwi.monetdb.mcl.net.MapiSocket.connect(MapiSocket.java:252) at nl.cwi.monetdb.jdbc.MonetConnection.<init>(MonetConnection.java:232) at nl.cwi.monetdb.jdbc.MonetDriver.connect(MonetDriver.java:171) at java.sql.DriverManager.getConnection(DriverManager.java:664) at java.sql.DriverManager.getConnection(DriverManager.java:247) Does anyone has any tips? The jdbc driver version used is 2.13.

2 1

Re: MonetDB: default - We were severely underquoting according to th...
by Niels Nes 01 Aug '16

01 Aug '16

Sjoerd Kunnen we wel voor perl-DBI testen? Nu draaien somige tests niet (geven errors) omdat perl-DBI mist. Niels On Mon, Aug 01, 2016 at 03:55:06PM +0200, Sjoerd Mullender wrote: > Changeset: a64440dcd8bf for MonetDB > URL: http://dev.monetdb.org/hg/MonetDB?cmd=changeset;node=a64440dcd8bf > Modified Files: > configure.ag > Branch: default > Log Message: > > We were severely underquoting according to the autoconf documentation. > > > diffs (truncated from 2554 to 300 lines): > > diff --git a/configure.ag b/configure.ag > --- a/configure.ag > +++ b/configure.ag > @@ -9,7 +9,7 @@ dnl Copyright 1997 - July 2008 CWI, Augu > > # Process this file with autoconf to produce a configure script. > > -AC_PREREQ(2.60) > +AC_PREREQ([2.60]) > AC_INIT([MonetDB], [11.24.0], [info(a)monetdb.org] [MonetDB], [http://www.monetdb.org/] > AC_CONFIG_AUX_DIR([buildtools/conf]) > AC_CONFIG_MACRO_DIR([buildtools/conf]) > @@ -22,13 +22,13 @@ AC_CONFIG_HEADERS([monetdb_config.h]) > AC_SUBST([CONFIG_H], [monetdb_config.h]) > > HOST=[$target] > -AC_DEFINE_UNQUOTED(HOST, "$HOST", [Host identifier]) > +AC_DEFINE_UNQUOTED([HOST], ["$HOST"], [Host identifier]) > > if test x"$target_os" = xcygwin; then > - AC_DEFINE(WIN32, 1, [Define on Microsoft Windows (also under Cygwin)]) > + AC_DEFINE([WIN32], 1, [Define on Microsoft Windows (also under Cygwin)]) > fi > -AM_CONDITIONAL(NOT_WIN32, test x"$target_os" != xcygwin) > -AM_CONDITIONAL(NATIVE_WIN32, false) > +AM_CONDITIONAL([NOT_WIN32], [test x"$target_os" != xcygwin]) > +AM_CONDITIONAL([NATIVE_WIN32], [false]) > > m4_ifndef([PKG_PROG_PKG_CONFIG], > [m4_fatal([Could not locate the pkg-config autoconf > @@ -57,16 +57,16 @@ AC_DEFUN([MONETDB_VERSION_TO_NUMBER], > # except set the variables. The command to set them in case we do > # need to escape the \'s is Qvar=`echo "$var" | sed 's/\\\\/\\\\\\\\/g'` > BUILD=[`pwd`] > -AC_SUBST(BUILD) > +AC_SUBST([BUILD]) > > SOURCE=[`(cd $srcdir && pwd)`] > -AC_SUBST(SOURCE) > +AC_SUBST([SOURCE]) > > . $srcdir/libversions > -AC_SUBST(GDK_VERSION) > -AC_SUBST(MAPI_VERSION) > -AC_SUBST(MONETDB5_VERSION) > -AC_SUBST(STREAM_VERSION) > +AC_SUBST([GDK_VERSION]) > +AC_SUBST([MAPI_VERSION]) > +AC_SUBST([MONETDB5_VERSION]) > +AC_SUBST([STREAM_VERSION]) > > # if no --prefix option, we must set Qprefix to the default value > # the problem is that prefix is set very late in the process if it > @@ -77,14 +77,14 @@ else > Qprefix="${prefix}" > fi > Qprefix="`echo "$Qprefix" | sed 's/\\\\/\\\\\\\\/g'`" > -AC_SUBST(Qprefix) > +AC_SUBST([Qprefix]) > # exec_prefix is created very late, so let make do the expansion > if test "${exec_prefix}x" = "NONEx"; then > Qexec_prefix='${prefix}' > else > Qexec_prefix="`echo "$exec_prefix" | sed 's/\\\\/\\\\\\\\/g'`" > fi > -AC_SUBST(Qexec_prefix) > +AC_SUBST([Qexec_prefix]) > > if test -f "$srcdir"/vertoo.data; then > dnl Developers compiling from Mercurial: > @@ -124,21 +124,21 @@ case "$CC" in > ;; > esac > > -AC_DEFINE(MONETDB_RELEASE, "unreleased", [Release name or "unreleased"]) > +AC_DEFINE([MONETDB_RELEASE], ["unreleased"], [Release name or "unreleased"]) > > dft_gdk=yes > -AC_ARG_ENABLE(gdk, > - AS_HELP_STRING([--enable-gdk], > - [enable support for GDK (default=yes)]), > - enable_gdk=$enableval, > - enable_gdk=$dft_gdk) > +AC_ARG_ENABLE([gdk], > + [AS_HELP_STRING([--enable-gdk], > + [enable support for GDK (default=yes)])], > + [enable_gdk=$enableval], > + [enable_gdk=$dft_gdk]) > > dft_monetdb5=yes > -AC_ARG_ENABLE(monetdb5, > - AS_HELP_STRING([--enable-monetdb5], > - [enable support for MonetDB5 (default=yes)]), > - enable_monetdb5=$enableval, > - enable_monetdb5=$dft_monetdb5) > +AC_ARG_ENABLE([monetdb5], > + [AS_HELP_STRING([--enable-monetdb5], > + [enable support for MonetDB5 (default=yes)])], > + [enable_monetdb5=$enableval], > + [enable_monetdb5=$dft_monetdb5]) > case "$enable_gdk-$enable_monetdb5" in > no-yes) > AC_MSG_ERROR([MonetDB5 requires GDK.]) > @@ -150,32 +150,32 @@ no-auto) > esac > > dft_fits=auto > -AC_ARG_ENABLE(fits, > - AS_HELP_STRING([--enable-fits], > - [enable support for FITS (default=auto)]), > - enable_fits=$enableval, > - enable_fits=$dft_fits) > +AC_ARG_ENABLE([fits], > + [AS_HELP_STRING([--enable-fits], > + [enable support for FITS (default=auto)])], > + [enable_fits=$enableval], > + [enable_fits=$dft_fits]) > > dft_netcdf=auto > -AC_ARG_ENABLE(netcdf, > - AS_HELP_STRING([--enable-netcdf], > - [enable support for netcdf (default=auto)]), > - enable_netcdf=$enableval, > - enable_netcdf=$dft_netcdf) > +AC_ARG_ENABLE([netcdf], > + [AS_HELP_STRING([--enable-netcdf], > + [enable support for netcdf (default=auto)])], > + [enable_netcdf=$enableval], > + [enable_netcdf=$dft_netcdf]) > > dft_lidar=auto > -AC_ARG_ENABLE(lidar, > - AS_HELP_STRING([--enable-lidar], > - [enable support for LiDAR data (default=auto)]), > - have_lidar=$enableval, > - have_lidar=$dft_lidar) > +AC_ARG_ENABLE([lidar], > + [AS_HELP_STRING([--enable-lidar], > + [enable support for LiDAR data (default=auto)])], > + [have_lidar=$enableval], > + [have_lidar=$dft_lidar]) > > dft_sql=yes > -AC_ARG_ENABLE(sql, > - AS_HELP_STRING([--enable-sql], > - [enable support for MonetDB/SQL (default=yes)]), > - enable_sql=$enableval, > - enable_sql=$dft_sql) > +AC_ARG_ENABLE([sql], > + [AS_HELP_STRING([--enable-sql], > + [enable support for MonetDB/SQL (default=yes)])], > + [enable_sql=$enableval], > + [enable_sql=$dft_sql]) > case "$enable_monetdb5-$enable_sql" in > no-yes) > AC_MSG_ERROR([MonetDB/SQL requires MonetDB5.]) > @@ -187,11 +187,11 @@ no-auto) > esac > > dft_geom=auto > -AC_ARG_ENABLE(geom, > - AS_HELP_STRING([--enable-geom], > - [enable support for geom module (default=auto)]), > - enable_geom=$enableval, > - enable_geom=$dft_geom) > +AC_ARG_ENABLE([geom], > + [AS_HELP_STRING([--enable-geom], > + [enable support for geom module (default=auto)])], > + [enable_geom=$enableval], > + [enable_geom=$dft_geom]) > case "$enable_monetdb5-$enable_geom" in > no-yes) > AC_MSG_ERROR([geom module requires MonetDB5]) > @@ -203,11 +203,11 @@ no-auto) > esac > > dft_shp=auto > -AC_ARG_ENABLE(shp, > - AS_HELP_STRING([--enable-shp], > - [enable support for ESRI Shapefiles (default=auto)]), > - have_shp=$enableval, > - have_shp=$dft_shp) > +AC_ARG_ENABLE([shp], > + [AS_HELP_STRING([--enable-shp], > + [enable support for ESRI Shapefiles (default=auto)])], > + [have_shp=$enableval], > + [have_shp=$dft_shp]) > case "$enable_geom-$have_shp" in > no-yes) > AC_MSG_ERROR([ESRI Shapefile vault requires the geom module]) > @@ -219,85 +219,85 @@ no-auto) > esac > > dft_mapi=yes > -AC_ARG_ENABLE(mapi, > - AS_HELP_STRING([--enable-mapi], > - [Enable MAPI (default=yes)]), > - enable_mapi=$enableval, > - enable_mapi=$dft_mapi) > -AM_CONDITIONAL(HAVE_MAPI, test x$enable_mapi != xno) > +AC_ARG_ENABLE([mapi], > + [AS_HELP_STRING([--enable-mapi], > + [Enable MAPI (default=yes)])], > + [enable_mapi=$enableval], > + [enable_mapi=$dft_mapi]) > +AM_CONDITIONAL([HAVE_MAPI], [test x$enable_mapi != xno]) > if test "x$enable_mapi" != xno; then > - AC_DEFINE(HAVE_MAPI, 1, [Enable MAPI]) > + AC_DEFINE([HAVE_MAPI], 1, [Enable MAPI]) > fi > > dft_embedded=no > -AC_ARG_ENABLE(embedded, > - AS_HELP_STRING([--enable-embedded], > - [enable support for running MonetDB as a library (default=no)]), > - enable_embedded=$enableval, > - enable_embedded=$dft_embedded) > -AM_CONDITIONAL(HAVE_EMBEDDED, test x$enable_embedded != xno) > +AC_ARG_ENABLE([embedded], > + [AS_HELP_STRING([--enable-embedded], > + [enable support for running MonetDB as a library (default=no)])], > + [enable_embedded=$enableval], > + [enable_embedded=$dft_embedded]) > +AM_CONDITIONAL([HAVE_EMBEDDED], [test x$enable_embedded != xno]) > if test "x$enable_embedded" != xno; then > - AC_DEFINE(HAVE_EMBEDDED, 1, [Support for MonetDB as a library]) > + AC_DEFINE([HAVE_EMBEDDED], 1, [Support for MonetDB as a library]) > fi > > dft_embedded_r=no > -AC_ARG_ENABLE(embedded-r, > - AS_HELP_STRING([--enable-embedded-r], > - [enable support for running MonetDB as a library for R (default=no)]), > - enable_embedded_r=$enableval, > - enable_embedded_r=$dft_embedded_r) > -AM_CONDITIONAL(HAVE_EMBEDDED_R, test x$enable_embedded_r != xno) > +AC_ARG_ENABLE([embedded-r], > + [AS_HELP_STRING([--enable-embedded-r], > + [enable support for running MonetDB as a library for R (default=no)])], > + [enable_embedded_r=$enableval], > + [enable_embedded_r=$dft_embedded_r]) > +AM_CONDITIONAL([HAVE_EMBEDDED_R], [test x$enable_embedded_r != xno]) > if test "x$enable_embedded_r" != xno; then > - AC_DEFINE(HAVE_EMBEDDED_R, 1, [Support for MonetDB as a library for R]) > + AC_DEFINE([HAVE_EMBEDDED_R], 1, [Support for MonetDB as a library for R]) > fi > > dft_gsl=auto > -AC_ARG_ENABLE(gsl, > - AS_HELP_STRING([--enable-gsl], > - [enable support for GSL (default=auto)]), > - enable_gsl=$enableval, > - enable_gsl=$dft_gsl) > +AC_ARG_ENABLE([gsl], > + [AS_HELP_STRING([--enable-gsl], > + [enable support for GSL (default=auto)])], > + [enable_gsl=$enableval], > + [enable_gsl=$dft_gsl]) > > dft_rintegration=auto > -AC_ARG_ENABLE(rintegration, > - AS_HELP_STRING([--enable-rintegration], > - [enable support for R integration into MonetDB (default=auto)]), > - enable_rintegration=$enableval, > - enable_rintegration=$dft_rintegration) > +AC_ARG_ENABLE([rintegration], > + [AS_HELP_STRING([--enable-rintegration], > + [enable support for R integration into MonetDB (default=auto)])], > + [enable_rintegration=$enableval], > + [enable_rintegration=$dft_rintegration]) > > dft_pyintegration=auto > -AC_ARG_ENABLE(pyintegration, > - AS_HELP_STRING([--enable-pyintegration], > - [enable support for Python integration into MonetDB (default=auto)]), > - enable_pyintegration=$enableval, > - enable_pyintegration=$dft_pyintegration) > +AC_ARG_ENABLE([pyintegration], > + [AS_HELP_STRING([--enable-pyintegration], > + [enable support for Python integration into MonetDB (default=auto)])], > + [enable_pyintegration=$enableval], > + [enable_pyintegration=$dft_pyintegration]) > > dft_odbc=auto > -AC_ARG_ENABLE(odbc, > - AS_HELP_STRING([--enable-odbc], > - [compile the MonetDB ODBC driver (default=auto)]), > - enable_odbc=$enableval, > - enable_odbc=$dft_odbc) > +AC_ARG_ENABLE([odbc], > + [AS_HELP_STRING([--enable-odbc], > + [compile the MonetDB ODBC driver (default=auto)])], > + [enable_odbc=$enableval], > + [enable_odbc=$dft_odbc]) > > dft_testing=auto > -AC_ARG_ENABLE(testing, > - AS_HELP_STRING([--enable-testing], > - [enable support for testing (default=auto)]), > - enable_testing=$enableval, > _______________________________________________ > checkin-list mailing list > checkin-list(a)monetdb.org > https://www.monetdb.org/mailman/listinfo/checkin-list -- Niels Nes, Manager ITF, Centrum Wiskunde & Informatica (CWI) Science Park 123, 1098 XG Amsterdam, The Netherlands room L3.14, phone ++31 20 592-4098 sip:4098@sip.cwi.nl url: https://www.cwi.nl/people/niels e-mail: Niels.Nes(a)cwi.nl

1 0

Parameters in bind function
by Narendra Pradeep 01 Aug '16

01 Aug '16

Hello everyone , I am trying to understand the MAL statements for joining three tables . On which I came across bind function in MAL statement . I have a table lineitem with l_quantity in it . So what does the following statements means 1 ) ( X_94:bat[:oid,:oid] ,X_95:bat[:oid,:int] ) := sql.bind(X_4,"sys","lineitem","l_quantity",2,0,4); 2 ) (X_96:bat[:oid,:oid] ,X_97:bat[:oid,:int] ) := sql.bind(X_4,"sys","lineitem","l_quantity",2,1,4); what does the 5 th and 6 th argument means ?

1 0

does cracking implemented in monetdb
by Narendra Pradeep 28 Jul '16

28 Jul '16

My questions are 1) Does cracking implemented in monetdb ? 2) Which file contains the code for cracking in source ? 3) Can we trace cracking and find whether query reads record from disc or the cracked columns ? Thanks in advance. regards , Pradeep.

2 1

Re: MonetDB: Jun2016 - Trace BATs should be reset before the next TR...
by Sjoerd Mullender 28 Jul '16

28 Jul '16

This is not correct. I made the exact opposite change in changeset 080dcfcb43e4 in order to fix memory leaks. That worked, but now those leaks are back. Clearly another solution is needed. On 28/07/16 00:31, Martin Kersten wrote: > Changeset: 25c896ae6a53 for MonetDB > URL: http://dev.monetdb.org/hg/MonetDB?cmd=changeset;node=25c896ae6a53 > Modified Files: > monetdb5/mal/mal_profiler.c > Branch: Jun2016 > Log Message: > > Trace BATs should be reset before the next TRACE call > BBPunfix was slipped in, while it should simply be a BATclear. > > > diffs (36 lines): > > diff --git a/monetdb5/mal/mal_profiler.c b/monetdb5/mal/mal_profiler.c > --- a/monetdb5/mal/mal_profiler.c > +++ b/monetdb5/mal/mal_profiler.c > @@ -761,19 +761,19 @@ clearTrace(void) > return; /* not initialized */ > } > /* drop all trace tables */ > - BBPunfix(TRACE_id_event->batCacheid); > - BBPunfix(TRACE_id_time->batCacheid); > - BBPunfix(TRACE_id_pc->batCacheid); > - BBPunfix(TRACE_id_thread->batCacheid); > - BBPunfix(TRACE_id_ticks->batCacheid); > - BBPunfix(TRACE_id_rssMB->batCacheid); > - BBPunfix(TRACE_id_tmpspace->batCacheid); > - BBPunfix(TRACE_id_inblock->batCacheid); > - BBPunfix(TRACE_id_oublock->batCacheid); > - BBPunfix(TRACE_id_minflt->batCacheid); > - BBPunfix(TRACE_id_majflt->batCacheid); > - BBPunfix(TRACE_id_nvcsw->batCacheid); > - BBPunfix(TRACE_id_stmt->batCacheid); > + BBPclear(TRACE_id_event->batCacheid); > + BBPclear(TRACE_id_time->batCacheid); > + BBPclear(TRACE_id_pc->batCacheid); > + BBPclear(TRACE_id_thread->batCacheid); > + BBPclear(TRACE_id_ticks->batCacheid); > + BBPclear(TRACE_id_rssMB->batCacheid); > + BBPclear(TRACE_id_tmpspace->batCacheid); > + BBPclear(TRACE_id_inblock->batCacheid); > + BBPclear(TRACE_id_oublock->batCacheid); > + BBPclear(TRACE_id_minflt->batCacheid); > + BBPclear(TRACE_id_majflt->batCacheid); > + BBPclear(TRACE_id_nvcsw->batCacheid); > + BBPclear(TRACE_id_stmt->batCacheid); > TRACE_init = 0; > MT_lock_unset(&mal_contextLock); > initTrace(); > _______________________________________________ > checkin-list mailing list > checkin-list(a)monetdb.org > https://www.monetdb.org/mailman/listinfo/checkin-list > -- Sjoerd Mullender

1 0

Instrumentation in monetdb
by Narendra Pradeep 22 Jul '16

22 Jul '16

eello everyone , How can I get instumemtation in monetdb ? I need thedetails like planning time , execution time for each MAL statement in monetdb . Is there any command for it ? regards , Pradeep

2 1

Fwd: Monet php_mapi.inc
by Mark Raasveldt 22 Jul '16

22 Jul '16

> Begin forwarded message: > > From: Alberto Ferrari <aferrari(a)starconnecting.com> > Subject: Monet php_mapi.inc > Date: 21 July 2016 at 17:22:34 GMT+2 > To: Mark Raasveldt <m.raasveldt(a)cwi.nl> > > Hi Mark, > (about https://dev.monetdb.org/hg/MonetDB/file/tip/clients/php/lib/php_mapi.inc) > > I've been using the "php_mapi.inc" in my php web application in the > last few months. So I modified the "php_mapi.inc" to a php class, > which is more efficient and correct (to me). > I am not registered in the Mercurial repo, so , if you (or somebody) > are interested on my "class" version, I can send you attached, and > then feel free to publish it, if you want. > Just tell me. > > regards > Alberto Ferrari.

2 1

Announcement: New Jun2016-SP1 Bugfix release of MonetDB suite
by Sjoerd Mullender 22 Jul '16

22 Jul '16

The MonetDB team at CWI/MonetDB BV is pleased to announce the Jun2016-SP1 bugfix release of the MonetDB suite of programs. More information about MonetDB can be found on our website at <http://www.monetdb.org/>. For details on this release, please see the release notes at <http://www.monetdb.org/Downloads/ReleaseNotes>. As usual, the download location is <http://dev.monetdb.org/downloads/>. Jun 2016-SP1 bugfix release . * Lots of memory leaks have been plugged across the whole system. Java Module * Corrected PROCEDURE_TYPE output value of method DatabaseMetaData.getProcedures(). It used to return procedureReturnsResult. Now it returns procedureNoResult. Corrected ORDINAL_POSITION output value of method DatabaseMetaData.getProcedureColumns(). It used to start with 0, but as procedures do not return a result value it now starts with 1 for all the procedure arguments, as defined by the JDBC API. * Improved output of method DatabaseMetaData.getProcedures(). The REMARKS column now contains the procedure definition as stored in sys.functions.func. The SPECIFIC_NAME column now contains the procedure unique identifier as stored in sys.functions.id. This allows the caller to retrieve the specific overloaded procedure which has the same name, but different arguments. Also improved output of method DatabaseMetaData.getProcedureColumns(). The SPECIFIC_NAME column now contains the procedure unique identifier as stored in sys.functions.id. This allows the caller to retrieve the proper arguments of the specific overloaded procedure by matching the SPECIFIC_NAME value. * Improved output of method DatabaseMetaData.getFunctions(). The REMARKS column now contains the function definition as stored in sys.functions.func. The SPECIFIC_NAME column now contains the function unique identifier as stored in sys.functions.id. This allows the caller to retrieve the specific overloaded function which has the same name, but different arguments. Also improved output of method DatabaseMetaData.getFunctionColumns(). The SPECIFIC_NAME column now contains the function unique identifier as stored in sys.functions.id. This allows the caller to retrieve the proper arguments of the specific overloaded function by matching the SPECIFIC_NAME value. Bug Fixes * 4014: KILL signal * 4021: Analyze query does not escape input [security] * 4026: JDBC driver incorrectly converts TINYINT fields to String instead of an integer type. * 4028: inputs not the same size * 4031: mclient doesn't accept - argument to refer to stdin * 4032: no decimal places after update. ODBC driver * 4035: SQL Function call bug * 4036: Possible sql_catalog corruption due to unclean backuped tail

1 0

Tracking query & user information during execution
by Playfair, Daniel 19 Jul '16

19 Jul '16

Hi MonetDB Developers, I'm working to add intra-query checkpoint/restart functionality to MonetDB as part of my PhD work. I have a partial implementation and have completed the component for transmitting intermediate values (to a separate middleware), and have a plan that I think should work for replay. Most of these modifications are in the runMALsequence function in the mal interpreter. Currently my implementation transmits all intermediates and I'm having some difficulty on the component to selectively transmit selected intermediates. I was hoping you might be able answer some of my questions to point me in the right direction on this aspect. I intend for the actual checkpoint planning to be performed on the query plan graphs in an offline step. This relates to one of my problems - The MAL variable names for intermediates seem to change accross executions. 1. Is there a way to have these identifiers be consistent? 2. What factors determine the values they take? Secondly, I am having difficulty with tracking/accessing information about the user, original SQL query, and current MAL operation step to be able to tag the transmitted intermediates with sufficient data for the middleware & load balancer to be able to relate this to a user session & query. I believe this should be accessible via 3. How can I access this information from the inputs available in runMALsequence (in mal_interpreter.c) I believe it's probably somewhere in the mal stack, instruction record, or client structs, but I am having difficulty here. Any help and input would be greatly appreciated. Many thanks, ~ Daniel.

1 0

Announcement: New Jun2016 Feature release of MonetDB suite
by Sjoerd Mullender 23 Jun '16

23 Jun '16

The MonetDB team at CWI/MonetDB BV is pleased to announce the Jun2016 feature release of the MonetDB suite of programs. More information about MonetDB can be found on our website at <http://www.monetdb.org/>. For details on this release, please see the release notes at <http://www.monetdb.org/Downloads/ReleaseNotes>. As usual, the download location is <http://dev.monetdb.org/downloads/>. Jun 2016 feature release MonetDB5 Server * Extended the storage() table producing function to also accept storage([schemaname [, tablename [, columnname]]]) * The :bat[:oid,:any] type descriptor has been turned into its columnar version :bat[:any] * Converted the *.mal scripts into *.malC versions to prepare for removal of the mserver console. * Removed functions mat.hasMoreElements, mat.info, mat.mergepack, mat. newIterator, mat.project, mat.pack2, mat.sortReverse, mat.sort, and mat.slice. * Removed grouped aggregate functions from the aggr module in which the groups were indicated by the head column of the bat to be aggregated. Use the interface with a separate group bat instead. * The server now stops executing a query when the client disconnects. * Removed algebra.join. Use algebra.subjoin instead. * Removed algebra.antijoin. Use algebra.subantijoin or algebra.subthetajoin instead. * The MAL function 'leftfetchjoin' is renamed to its relational algebra version 'projection' * The generic property handling scheme has been removed. It was used in just a few places, for widely different purposes and contained cruft. * Removed functions str.iconv and str.codeset. Internally, strings are always in UTF-8 encoding, so we can't allow code set conversions. * Removed algebra.like with a BAT argument. Use algebra.likesubselect instead. * Removed algebra.leftjoin. Use algebra.subleftjoin or algebra.leftfetchjoin instead. * Removed algebra.tdiff and algebra.tinter. * Removed algebra.sample. Use sample.subuniform instead. * Removed algebra.select. Use algebra.subselect instead. * Removed algebra.revert. * Removed bat.order and bat.orderReverse functions. * Changed client.getUsers to return two bats, one with the user id and one with the user name. * Implemented algebra.subdiff and algebra.subinter. * Removed algebra.tdifference and algebra.tintersect. * Removed algebra.tunion. Build Environment * A new package MonetDB-lidar (Fedora) or libmonetdb5-server-lidar (Debian/Ubuntu) has been created to work with LiDAR data. * Implemented a systemd configuration file for a monetdbd.service on systems that support it (Fedora, newer Ubuntu). Client Package * The Ruby client is now in a separate repository (http://dev.monetdb.org/hg/monetdb-ruby) and released independently. Geom Module * The geom module has been completely overhauled. Types are now specified as GEOMETRY(POINT) instead of POINT, old functions have been removed, new ones introduced. However, even with all the changes, a database upgrade should still be possible (as always, make a backup first). MonetDB Common * Removed BATconst. Use BATconstant instead. * Changed BATconstant. It now has a new first argument with the seqbase for the head column. * Removed BATmmap. It was no longer used. * BUNdelete and BATdel don't accept a foce argument and only allow deleting values that have not yet been committed. BUNdelete exchanges the deleted value with the last value (if the deleted value isn't the last value). BATdel compacts the BAT by shifting values after the deleted values up. The list of to-be-deleted values in BATdel must be sorted and unique. * Removed BUNreplace from list of exported functions. It wasn't used, and there is still BUNinplace and void_inplace that do more-or-less the same thing. * Changed BATderiveHeadProps to BATderiveTailProps (and it now works on the tail column). * Removed unused functions BATalpha, BATdelta, and BATprev. * Removed function VIEWcombine. Use BATdense instead. * Removed "left" parameter from BUNinplace. It wasn't used since the BAT it works on is dense headed. * Created function BATdense to easily create a [void,void] BAT with specified seqbases for head and tail and a count. * Removed function BATmark. When all head columns are dense, BATmark basically only created a new [void,void] BAT. * Renamed BATsubsort to BATsort. * Removed "sub" from the names of the function BATsubselect, BATthetasubselect, BATsubcross, BATsubleftjoin, BATsubouterjoin, BATsubthetajoin, BATsubsemijoin, BATsubdiff, BATsubjoin, BATsubbandjoin, BATsubrangejoin, and BATsubunique. * Removed BATsubleftfetchjoin: it was not used. * Removed BATcross1. Use BATsubcross instead. * Removed all versions of the SORTloop macro. * Removed Batkdiff. Use BATsubdiff instead. * Removed BATselect. Use BATsubselect instead. * Removed BATsemijoin. Use BATsubsemijoin instead. * Removed BATjoin. Use BATsubjoin instead. * Removed BATleftjoin. Use BATsubleftjoin or BATproject instead. * Removed BATleftfetchjoin. Use BATproject instead. * Removed BUNins from the list of exported functions. * Removed legacy functions BATuselect and BATuselect_. * Removed legacy functions BATantijoin, BATbandjoin, BATouterjoin, BATrangejoin, and BATthetajoin. * Removed function BATrevert. * BATordered now works on the TAIL column. * Removed obsolete functions BATorder() and BATorder_rev(). * Implemented BATsubdiff which returns a list of OIDs (sorted, i.e. usable as candidate list) of tuples in the left input whose value does not occur in the right input. * Removed function BATkintersect. It wasn't used anymore. It's functionality can be obtained by using BATsubsemijoin. * Removed the function BATkunion. Java Module * Fixed problem in DatabaseMetaData.getUDTs() when it was called with types parameter filled. It used to throw SQException with message: SELECT: identifier 'DATA_TYPE' unknown. Now it returns the UDTs which match the provided array of data types. * Implemented MonetDatabaseMetaData.supportsConvert() and MonetDatabaseMetaData.supportsConvert(int fromType, int toType) methods. It used to always return false. Now it returns true for the supported conversions. This fixes Bug 3460. * ResultSet.setFetchSize(): added a dummy implementation to get rid of the SQLFeatureNotSupportedException. In MonetDB, it does not make sense to set the fetch size of a result set. If one really wants to set the fetch size, one should use Statement.setFetchSize() instead. * Fixed resource leak in ResultSetMetaData. It created and cached a ResultSet object for each column but never closed the ResultSet objects. * Corrected DatabaseMetaData methods which accept a catalog filter argument. Those methods will now filter the results on the specified catalog name, whereas previously the catalog filter argument was ignored. * Corrected output of column KEY_SEQ of DatabaseMetaData methods: getPrimaryKeys(), getImportedKeys(), getExportedKeys() and getCrossReference(). It now starts at 1 instead of 0 previously. * Corrected DatabaseMetaData.getSchemas() by returning 2 instead of 3 columns. * Improved DatabaseMetaData.getColumns() by returning two additional columns: IS_AUTOINCREMENT and IS_GENERATEDCOLUMN. * Improved DatabaseMetaData.getTypeInfo(). It now returns better information on LITERAL_PREFIX, LITERAL_SUFFIX, CREATE_PARAMS, CASE_SENSITIVE, FIXED_PREC_SCALE and MAXIMUM_SCALE for some data types. Also the returned rows are now ordered by DATA_TYPE, TYPE_NAME, PRECISION as required by the specs. Also corrected output column names "searchable" into "SEARCHABLE" and "MAXIMUM SCALE" into "MAXIMUM_SCALE" to match the JDBC specification. * Corrected DatabaseMetaData.getPseudoColumns(). It used to return 12 empty rows. Now it returns no rows as MonetDB does not have pseudo columns. * Implemented method DatabaseMetaData.getClientProperties(). It used to always return a resultset with 4 completely empty rows. It now returns a resultset with the possible connection properties. * Implemented method DatabaseMetaData.getUDTs(). It used to return an empty resultset. Now it returns the User Defined Types such as inet, json, url and uuid. * Corrected the returned table types in DatabaseMetaData.getTableTypes(). It now returns all 10 table types (as stored in sys.table_types) instead of the previously 8 hardcoded table types. For old MonetDB servers which do not have the sys.table_types table, the old behavior is retained. * Implemented methods DatabaseMetadata.getProcedures() and DatabaseMetadata.getProcedureColumns(). They used to return an empty resultset. Now they return the expected Procedures and ProcedureColumns. Also getProcedureColumns() now returns a resultset with all 20 columns instead of 13 columns previously. * Method getFunctionColumns() in DatabaseMetadata used to throw an SQLException: getFunctionColumns(String, String, String, String) is not implemented. This method is now implemented and returns a resultset. * Method getFunctions() in DatabaseMetadata used to throw an SQLException: SELECT: no such column 'functions.sql' This has been corrected. It now returns a resultset as requested. * The resultsets of DatabaseMetadata methods now no longer return a value for the *_CAT columns as MonetDB does not support Catalogs. * Fixed a memory leak in MonetDatabaseMetaData.java for a static cache which kept references to closed Connection objects. * Improved JdbcClient program when presenting query data to console. It used to send an SQL catalog query for each query result column which slowed down the interactive response considerably. These additional SQL catalog queries have been eliminated. * Corrected MonetResultSet.getObject(String columnName). It no longer throws a NullPointerException in cases where internally a MonetVirtualResultSet is used. * Fixed Connection.isValid(): this method should never attempt to close the connection, even if an error has occurred. SQL * Disallow GRANT <some-user> TO <role-or-use>. Only explicitly created roles can be granted. * Extended grantable schema privileges: when a user is granted the "sysadmin" role, the user now hcan not only create schemas, but also drop schemas. * Introduced COPY INTO/ COPY FROM global privileges. These privileges allows other users than 'monetdb' to be granted the privileges to use the MonetDB bulk data loading/exporting features, i.e., COPY FROM <file> and COPY INTO <file>. Bug Fixes * 2407: Merovingian: allow binds to given ip/interface * 2815: No SRID support * 3361: constants as MAL function parameters prevent intermediate reuse * 3374: UNIQUE constraint does not set tkey property on the corresponding BAT * 3460: incomplete implementation of JDBC driver supportsConvert(), supportsConvert(int fromType, int toType) methods in MonetDatabaseMetaData.java * 3711: JDBC connection using jdbcclient.jar is very slow * 3877: MonetDBLite should allow close then re-open databases? without restarting R * 3911: Invalid connect() call in 'redirect' mode * 3920: query with DISTINCT and correlated scalar subquery in SELECT list causes Assertion failure and crash of mserver5 * 3927: COUNT( distinct my_clob ) , QUANTILE( my_double , number ) fails in dev build * 3956: MonetDBLite unable to execute LIMIT 1 statement * 3972: Drastic Memory leak of 600GBs while generating plan for Query with 25 joins * 3974: Prepared statement rel_bin.c:2378: rel2bin_project: Assertion `0' failed. * 3975: Suspicious code in store_manager() on exit path * 3978: SQL returns TypeException 'aggr.subcorr' undefined for sys.corr function * 3980: JOIN with references on both sides crashes mserver * 3981: Incorrect LEFT JOIN when FROM clause contains nested subqueries * 3983: Creation of a Foreign Key which partially maps to a primary key is accepted without a warning * 3984: Multiple paths in the .profile * 3985: ruby-monetdb-sql gem fails for negative timezone offset (USA, etc.) * 3987: Segfault on malformed csv import * 3991: MonetDBLite feature request: default monetdb.sequential to FALSE on windows * 3994: MonetDBLite dbDisconnect with shutdown=TRUE freezes the console on windows * 3995: NullPointerException when calling getObject() * 3997: calling scalar functions sys.isaUUID(str) or sys.isaUUID(uuid) fail * 3999: length() returns wrong length for strings which have spaces at the end of the string * 4010: RELEASE SAVEPOINT after ALTER TABLE crashes mserver5 * 4011: sys.sessions.user column always shows 'monetdb' * 4013: GDKextendf does not free up memory when it fails due to insufficient resources * 4015: Daemon crashes on database release command

1 0

Announcement: New Jun2016 Feature release of MonetDB suite
by Sjoerd Mullender 23 Jun '16

23 Jun '16

The MonetDB team at CWI/MonetDB BV is pleased to announce the Jun2016 feature release of the MonetDB suite of programs. More information about MonetDB can be found on our website at <http://www.monetdb.org/>. For details on this release, please see the release notes at <http://www.monetdb.org/Downloads/ReleaseNotes>. As usual, the download location is <http://dev.monetdb.org/downloads/>. Jun 2016 feature release MonetDB5 Server * Extended the storage() table producing function to also accept storage([schemaname [, tablename [, columnname]]]) * The :bat[:oid,:any] type descriptor has been turned into its columnar version :bat[:any] * Converted the *.mal scripts into *.malC versions to prepare for removal of the mserver console. * Removed functions mat.hasMoreElements, mat.info, mat.mergepack, mat. newIterator, mat.project, mat.pack2, mat.sortReverse, mat.sort, and mat.slice. * Removed grouped aggregate functions from the aggr module in which the groups were indicated by the head column of the bat to be aggregated. Use the interface with a separate group bat instead. * The server now stops executing a query when the client disconnects. * Removed algebra.join. Use algebra.subjoin instead. * Removed algebra.antijoin. Use algebra.subantijoin or algebra.subthetajoin instead. * The MAL function 'leftfetchjoin' is renamed to its relational algebra version 'projection' * The generic property handling scheme has been removed. It was used in just a few places, for widely different purposes and contained cruft. * Removed functions str.iconv and str.codeset. Internally, strings are always in UTF-8 encoding, so we can't allow code set conversions. * Removed algebra.like with a BAT argument. Use algebra.likesubselect instead. * Removed algebra.leftjoin. Use algebra.subleftjoin or algebra.leftfetchjoin instead. * Removed algebra.tdiff and algebra.tinter. * Removed algebra.sample. Use sample.subuniform instead. * Removed algebra.select. Use algebra.subselect instead. * Removed algebra.revert. * Removed bat.order and bat.orderReverse functions. * Changed client.getUsers to return two bats, one with the user id and one with the user name. * Implemented algebra.subdiff and algebra.subinter. * Removed algebra.tdifference and algebra.tintersect. * Removed algebra.tunion. Build Environment * A new package MonetDB-lidar (Fedora) or libmonetdb5-server-lidar (Debian/Ubuntu) has been created to work with LiDAR data. * Implemented a systemd configuration file for a monetdbd.service on systems that support it (Fedora, newer Ubuntu). Client Package * The Ruby client is now in a separate repository (http://dev.monetdb.org/hg/monetdb-ruby) and released independently. Geom Module * The geom module has been completely overhauled. Types are now specified as GEOMETRY(POINT) instead of POINT, old functions have been removed, new ones introduced. However, even with all the changes, a database upgrade should still be possible (as always, make a backup first). MonetDB Common * Removed BATconst. Use BATconstant instead. * Changed BATconstant. It now has a new first argument with the seqbase for the head column. * Removed BATmmap. It was no longer used. * BUNdelete and BATdel don't accept a foce argument and only allow deleting values that have not yet been committed. BUNdelete exchanges the deleted value with the last value (if the deleted value isn't the last value). BATdel compacts the BAT by shifting values after the deleted values up. The list of to-be-deleted values in BATdel must be sorted and unique. * Removed BUNreplace from list of exported functions. It wasn't used, and there is still BUNinplace and void_inplace that do more-or-less the same thing. * Changed BATderiveHeadProps to BATderiveTailProps (and it now works on the tail column). * Removed unused functions BATalpha, BATdelta, and BATprev. * Removed function VIEWcombine. Use BATdense instead. * Removed "left" parameter from BUNinplace. It wasn't used since the BAT it works on is dense headed. * Created function BATdense to easily create a [void,void] BAT with specified seqbases for head and tail and a count. * Removed function BATmark. When all head columns are dense, BATmark basically only created a new [void,void] BAT. * Renamed BATsubsort to BATsort. * Removed "sub" from the names of the function BATsubselect, BATthetasubselect, BATsubcross, BATsubleftjoin, BATsubouterjoin, BATsubthetajoin, BATsubsemijoin, BATsubdiff, BATsubjoin, BATsubbandjoin, BATsubrangejoin, and BATsubunique. * Removed BATsubleftfetchjoin: it was not used. * Removed BATcross1. Use BATsubcross instead. * Removed all versions of the SORTloop macro. * Removed Batkdiff. Use BATsubdiff instead. * Removed BATselect. Use BATsubselect instead. * Removed BATsemijoin. Use BATsubsemijoin instead. * Removed BATjoin. Use BATsubjoin instead. * Removed BATleftjoin. Use BATsubleftjoin or BATproject instead. * Removed BATleftfetchjoin. Use BATproject instead. * Removed BUNins from the list of exported functions. * Removed legacy functions BATuselect and BATuselect_. * Removed legacy functions BATantijoin, BATbandjoin, BATouterjoin, BATrangejoin, and BATthetajoin. * Removed function BATrevert. * BATordered now works on the TAIL column. * Removed obsolete functions BATorder() and BATorder_rev(). * Implemented BATsubdiff which returns a list of OIDs (sorted, i.e. usable as candidate list) of tuples in the left input whose value does not occur in the right input. * Removed function BATkintersect. It wasn't used anymore. It's functionality can be obtained by using BATsubsemijoin. * Removed the function BATkunion. Java Module * Fixed problem in DatabaseMetaData.getUDTs() when it was called with types parameter filled. It used to throw SQException with message: SELECT: identifier 'DATA_TYPE' unknown. Now it returns the UDTs which match the provided array of data types. * Implemented MonetDatabaseMetaData.supportsConvert() and MonetDatabaseMetaData.supportsConvert(int fromType, int toType) methods. It used to always return false. Now it returns true for the supported conversions. This fixes Bug 3460. * ResultSet.setFetchSize(): added a dummy implementation to get rid of the SQLFeatureNotSupportedException. In MonetDB, it does not make sense to set the fetch size of a result set. If one really wants to set the fetch size, one should use Statement.setFetchSize() instead. * Fixed resource leak in ResultSetMetaData. It created and cached a ResultSet object for each column but never closed the ResultSet objects. * Corrected DatabaseMetaData methods which accept a catalog filter argument. Those methods will now filter the results on the specified catalog name, whereas previously the catalog filter argument was ignored. * Corrected output of column KEY_SEQ of DatabaseMetaData methods: getPrimaryKeys(), getImportedKeys(), getExportedKeys() and getCrossReference(). It now starts at 1 instead of 0 previously. * Corrected DatabaseMetaData.getSchemas() by returning 2 instead of 3 columns. * Improved DatabaseMetaData.getColumns() by returning two additional columns: IS_AUTOINCREMENT and IS_GENERATEDCOLUMN. * Improved DatabaseMetaData.getTypeInfo(). It now returns better information on LITERAL_PREFIX, LITERAL_SUFFIX, CREATE_PARAMS, CASE_SENSITIVE, FIXED_PREC_SCALE and MAXIMUM_SCALE for some data types. Also the returned rows are now ordered by DATA_TYPE, TYPE_NAME, PRECISION as required by the specs. Also corrected output column names "searchable" into "SEARCHABLE" and "MAXIMUM SCALE" into "MAXIMUM_SCALE" to match the JDBC specification. * Corrected DatabaseMetaData.getPseudoColumns(). It used to return 12 empty rows. Now it returns no rows as MonetDB does not have pseudo columns. * Implemented method DatabaseMetaData.getClientProperties(). It used to always return a resultset with 4 completely empty rows. It now returns a resultset with the possible connection properties. * Implemented method DatabaseMetaData.getUDTs(). It used to return an empty resultset. Now it returns the User Defined Types such as inet, json, url and uuid. * Corrected the returned table types in DatabaseMetaData.getTableTypes(). It now returns all 10 table types (as stored in sys.table_types) instead of the previously 8 hardcoded table types. For old MonetDB servers which do not have the sys.table_types table, the old behavior is retained. * Implemented methods DatabaseMetadata.getProcedures() and DatabaseMetadata.getProcedureColumns(). They used to return an empty resultset. Now they return the expected Procedures and ProcedureColumns. Also getProcedureColumns() now returns a resultset with all 20 columns instead of 13 columns previously. * Method getFunctionColumns() in DatabaseMetadata used to throw an SQLException: getFunctionColumns(String, String, String, String) is not implemented. This method is now implemented and returns a resultset. * Method getFunctions() in DatabaseMetadata used to throw an SQLException: SELECT: no such column 'functions.sql' This has been corrected. It now returns a resultset as requested. * The resultsets of DatabaseMetadata methods now no longer return a value for the *_CAT columns as MonetDB does not support Catalogs. * Fixed a memory leak in MonetDatabaseMetaData.java for a static cache which kept references to closed Connection objects. * Improved JdbcClient program when presenting query data to console. It used to send an SQL catalog query for each query result column which slowed down the interactive response considerably. These additional SQL catalog queries have been eliminated. * Corrected MonetResultSet.getObject(String columnName). It no longer throws a NullPointerException in cases where internally a MonetVirtualResultSet is used. * Fixed Connection.isValid(): this method should never attempt to close the connection, even if an error has occurred. SQL * Disallow GRANT <some-user> TO <role-or-use>. Only explicitly created roles can be granted. * Extended grantable schema privileges: when a user is granted the "sysadmin" role, the user now hcan not only create schemas, but also drop schemas. * Introduced COPY INTO/ COPY FROM global privileges. These privileges allows other users than 'monetdb' to be granted the privileges to use the MonetDB bulk data loading/exporting features, i.e., COPY FROM <file> and COPY INTO <file>. Bug Fixes * 2407: Merovingian: allow binds to given ip/interface * 2815: No SRID support * 3361: constants as MAL function parameters prevent intermediate reuse * 3374: UNIQUE constraint does not set tkey property on the corresponding BAT * 3460: incomplete implementation of JDBC driver supportsConvert(), supportsConvert(int fromType, int toType) methods in MonetDatabaseMetaData.java * 3711: JDBC connection using jdbcclient.jar is very slow * 3877: MonetDBLite should allow close then re-open databases? without restarting R * 3911: Invalid connect() call in 'redirect' mode * 3920: query with DISTINCT and correlated scalar subquery in SELECT list causes Assertion failure and crash of mserver5 * 3927: COUNT( distinct my_clob ) , QUANTILE( my_double , number ) fails in dev build * 3956: MonetDBLite unable to execute LIMIT 1 statement * 3972: Drastic Memory leak of 600GBs while generating plan for Query with 25 joins * 3974: Prepared statement rel_bin.c:2378: rel2bin_project: Assertion `0' failed. * 3975: Suspicious code in store_manager() on exit path * 3978: SQL returns TypeException 'aggr.subcorr' undefined for sys.corr function * 3980: JOIN with references on both sides crashes mserver * 3981: Incorrect LEFT JOIN when FROM clause contains nested subqueries * 3983: Creation of a Foreign Key which partially maps to a primary key is accepted without a warning * 3984: Multiple paths in the .profile * 3985: ruby-monetdb-sql gem fails for negative timezone offset (USA, etc.) * 3987: Segfault on malformed csv import * 3991: MonetDBLite feature request: default monetdb.sequential to FALSE on windows * 3994: MonetDBLite dbDisconnect with shutdown=TRUE freezes the console on windows * 3995: NullPointerException when calling getObject() * 3997: calling scalar functions sys.isaUUID(str) or sys.isaUUID(uuid) fail * 3999: length() returns wrong length for strings which have spaces at the end of the string * 4010: RELEASE SAVEPOINT after ALTER TABLE crashes mserver5 * 4011: sys.sessions.user column always shows 'monetdb' * 4013: GDKextendf does not free up memory when it fails due to insufficient resources * 4015: Daemon crashes on database release command

1 0

(no subject)
by Narendra Pradeep 22 Jun '16

22 Jun '16

While coming across Monetdb source there are three structures which were used most often.They are MalBlkPtr mb MalStkPtr stk InstrPtr pci What are they meant for ? what is their initial value and when do they get initialized ? Thanks

1 0

Announcement: New release candidate of MonetDB suite
by Sjoerd Mullender 14 Jun '16

14 Jun '16

-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA256 This is a heads up announcement. We at CWI and MonetDB have prepared a release candidate for the Jun2016 release. Usually we just foist a new release upon you, our unsuspecting users, but this time, because there are so many changes, we thought it better to give you a little warning. See http://dev.monetdb.org/downloads/testing/ for the release and the preliminary release notes [1]. Among the changes is a complete overhaul of the MonetDB-geom module. If you use that, make very sure you have your backups in order. Also note that since this new version depends on an up-to-date version of the geos library, there is no geom module available for older Linux releases (in particular, Debian 7 (wheezy), Ubuntu 12.04 LTS (Precise Pangolin) and Ubuntu 14.04 LTS (Trusty Tahr), and RedHat Enterprise Linux (CentOS and Scientific Linux) 6). The plan is to turn this release candidate into an actual release before the end of this month (June 2016). If you find a bug in this release candidate, please let us know on the bug tracker [2]. We will then determine whether to go ahead with the release (and hopefully fix the bug in the next bug-fix release) or postpone the release and fix the bug first. [1] https://dev.monetdb.org/downloads/testing/relnotes.html [2] http://bugs.monetdb.org (a.k.a. https://www.monetdb.org/bugzilla/) - -- Sjoerd Mullender -----BEGIN PGP SIGNATURE----- Version: GnuPG v2 iQEcBAEBCAAGBQJXX/EsAAoJEISMxT6LrWYg+AcIAIgIST/dUrL8fDjwp3/G41zc 4g9xWbSxIE/+nZLjlHM6Onr4sETceOVdDVuxUGoIbeu9+DZNQqEakUUrABrldvLx ie/kbNINtdntFvfF3W5Lyo/lWhel3IUZZhEXvsBT9JCSfg5kiLKYr/CPqw5QgOcU XgccIn7RAZYovfVoLo7jLxiMLeT9DJp74gy1OkjrIBwQ/GPVam/CzaO5Ungu0dRY cZiqoRhW1GEtUQHABq0O9J/gJcHfFbSlwDeQcnTouybO320gBRISYqgV8aTt0RdU /MVVkdOU3hLsShjAgWinRbQPNVwCLcZNAfxER9q4Z9tt15n/3E6WtPeKo2Bv030= =rnCK -----END PGP SIGNATURE-----

1 0

Announcement: New release candidate of MonetDB suite
by Sjoerd Mullender 14 Jun '16

14 Jun '16

-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA256 This is a heads up announcement. We at CWI and MonetDB have prepared a release candidate for the Jun2016 release. Usually we just foist a new release upon you, our unsuspecting users, but this time, because there are so many changes, we thought it better to give you a little warning. See http://dev.monetdb.org/downloads/testing/ for the release and the preliminary release notes [1]. Among the changes is a complete overhaul of the MonetDB-geom module. If you use that, make very sure you have your backups in order. Also note that since this new version depends on an up-to-date version of the geos library, there is no geom module available for older Linux releases (in particular, Debian 7 (wheezy), Ubuntu 12.04 LTS (Precise Pangolin) and Ubuntu 14.04 LTS (Trusty Tahr), and RedHat Enterprise Linux (CentOS and Scientific Linux) 6). The plan is to turn this release candidate into an actual release before the end of this month (June 2016). If you find a bug in this release candidate, please let us know on the bug tracker [2]. We will then determine whether to go ahead with the release (and hopefully fix the bug in the next bug-fix release) or postpone the release and fix the bug first. [1] https://dev.monetdb.org/downloads/testing/relnotes.html [2] http://bugs.monetdb.org (a.k.a. https://www.monetdb.org/bugzilla/) - -- Sjoerd Mullender -----BEGIN PGP SIGNATURE----- Version: GnuPG v2 iQEcBAEBCAAGBQJXX/EsAAoJEISMxT6LrWYg+AcIAIgIST/dUrL8fDjwp3/G41zc 4g9xWbSxIE/+nZLjlHM6Onr4sETceOVdDVuxUGoIbeu9+DZNQqEakUUrABrldvLx ie/kbNINtdntFvfF3W5Lyo/lWhel3IUZZhEXvsBT9JCSfg5kiLKYr/CPqw5QgOcU XgccIn7RAZYovfVoLo7jLxiMLeT9DJp74gy1OkjrIBwQ/GPVam/CzaO5Ungu0dRY cZiqoRhW1GEtUQHABq0O9J/gJcHfFbSlwDeQcnTouybO320gBRISYqgV8aTt0RdU /MVVkdOU3hLsShjAgWinRbQPNVwCLcZNAfxER9q4Z9tt15n/3E6WtPeKo2Bv030= =rnCK -----END PGP SIGNATURE-----

1 0

Updating for GCC 6
by Sahasranaman M S 24 May '16

24 May '16

Hi all, GCC 6.1 was relased late last month, and it was failing to build latest MonetDB, with the below 3 errors: stream.c:2116:24: error: logical ‘or’ of equal expressions [-Werror=logical-op] (errno == EAGAIN || errno == EWOULDBLOCK) && /* it was! */ ^~ stream.c:2142:24: error: logical ‘or’ of equal expressions [-Werror=logical-op] (errno == EAGAIN || errno == EWOULDBLOCK) ^~ mal_module.c: In function ‘getHelp’: mal_module.c:455:3: error: this ‘for’ clause does not guard... [-Werror=misleading-indentation] for(s= m->subscope[j]; s; s= s->peer) ^~~ mal_module.c:513:4: note: ...this statement, but the latter is misleadingly indented as if it is guarded by the ‘for’ m= m->sibling; ^ The first two appear to be because EAGAIN, EWOULDBLOCK have the same value on my machine (x86_64, Fedora 23 64bit). I don't know in what situation they would be different from each other, so I'm removing the check for EWOULDBLOCK and checking only for EAGAIN, as suggested by some on the Internet. The third error is due to bad indentation. If we reduce indentation of mal_module.c:513 by two levels, that solves it. It works for me with these changes, could someone confirm my solution to first two errors won't cause regressions, of suggest alternative solutions? Thanks, Sahas

2 6

Cache functions ?
by Brian Hood 16 May '16

16 May '16

Hi Guys, I've just noticed the cache functions in the mapi code what do these actually do and also look interesting ? mapi_export MapiMsg mapi_cache_limit(Mapi mid, int limit); mapi_export MapiMsg mapi_cache_shuffle(MapiHdl hdl, int percentage); mapi_export MapiMsg mapi_cache_freeup(MapiHdl hdl, int percentage); Regards, Brian Hood

1 0

Memory management - MonetDB on CentOS
by Vijay Krishna 07 May '16

07 May '16

Hi, We have been running tests on Memory management with the latest version of MonetDB on a CentOS installation. The machine has 1TB hard disk and 128 GB RAM space. The swap memory configured is 2GB. We were aware that MonetDB completely relies on the OS for Memory Management and is prone to OOM Killer crash and hence we wanted to simulate the crash. Hence we ran hundreds of threads executing various queries and noticed the following pattern of memory handling. 1. Initially, RAM was filled. 2. Upon RAM overflow, swap memory (2 GB) was used. 3. Upon swap overflow, the swap memory got extended and the 1TB memory that was free on the machine was used completely. 4. The OOM killer got invoked only when this 1TB memory got filled. 5. On consecutive executions, swap memory was started to be used even when RAM had 70GB of memory unused and free (not even used for cache files). 6. The /tmp partition was never used. Memory got overfilled only in the current partition. These observations led to the following questions about the memory management with CentOS. 1. Does MonetDB has no influence on the above behaviour and is it that only the OS is responsible for this? 2. Will the swap memory be expanded from the committed size by the OS if swap is running out of memory? In our case it expanded from 2GB to 1 TB. Our OS setting of overcommit is 0 (sysctl vm.overcommit_memory = 0) 3. When will the OOM Killer be invoked usually? Upon Swap fill or upon Hard disk space full? 4. Why was memory swapped even when there is 70GB free space in RAM? We tried reducing the swappiness to 10 (sysctl vm.swappiness=10), even then memory was swapped with 70 GB free space in RAM. 5. Does the OS writes intermediate results to /tmp directory? Or is it that MonetDB is bypassing this? 6. Is there any other OS configurations that we need to cross check regarding memory management? Any help much appreciated. Thanks in advance. Regards, Vijay.

1 0

Re: MonetDB: default - No need for IMPRINTS being a full-blown keyword.
by Lefteris 04 May '16

04 May '16

Sjoerd, there is a plan for imprints to become a keyword in SQL such that one may do the following: CREATE IMPRINTS INDEX idx_id ON table_id(column_id) just as ORDERED index. This is needed in many cases where for example Romulo wants to be able to create and drop imprints at will. On Wed, May 4, 2016 at 11:25 AM, Sjoerd Mullender <commits+sjoerd=acm.org(a)monetdb.org> wrote: > Changeset: 390e25b31196 for MonetDB > URL: http://dev.monetdb.org/hg/MonetDB?cmd=changeset;node=390e25b31196 > Modified Files: > sql/server/sql_parser.y > Branch: default > Log Message: > > No need for IMPRINTS being a full-blown keyword. > > > diffs (11 lines): > > diff --git a/sql/server/sql_parser.y b/sql/server/sql_parser.y > --- a/sql/server/sql_parser.y > +++ b/sql/server/sql_parser.y > @@ -5089,6 +5089,7 @@ non_reserved_word: > | TIME { $$ = sa_strdup(SA, "time"); } > | TIMESTAMP { $$ = sa_strdup(SA, "timestamp"); } > | INTERVAL { $$ = sa_strdup(SA, "interval"); } > +| IMPRINTS { $$ = sa_strdup(SA, "imprints"); } > > | PREPARE { $$ = sa_strdup(SA, "prepare"); } > | EXECUTE { $$ = sa_strdup(SA, "execute"); } > _______________________________________________ > checkin-list mailing list > checkin-list(a)monetdb.org > https://www.monetdb.org/mailman/listinfo/checkin-list

2 1

Monetdb Python Issue
by Patel, Nikunj (CONT) 26 Apr '16

26 Apr '16

Hi, I have created a simple table (create table dept (dept_id integer, dept_nm varchar(25)) and trying to insert records from file using Python and getting errors. All execute statements gives me same error message: connection = monetdb.sql.connect(username="monetdb", password="monetdb", hostname="10.81.102.71", port=2234, database="demo") cursor = connection.cursor() cursor.execute("COPY INTO dept FROM 'C:\AWS\did.csv' USING DELIMITERS;") cursor.execute("COPY INTO dept FROM 'C:\AWS\did.csv' USING DELIMITERS") cursor.execute("COPY INTO dept FROM STDIN;\n44444444,'AL'\n55555555,'JAFFRI'") Traceback (most recent call last): File "<stdin>", line 1, in <module> File "C:\Users\Q1\AppData\Local\Continuum\Anaconda2\lib\site-packages\monetdb\sql\cursors.py", line 175, in execute block = self.connection.execute(query) File "C:\Users\Q1\AppData\Local\Continuum\Anaconda2\lib\site-packages\monetdb\sql\connections.py", line 138, in execute return self.command('s' + query + ';') File "C:\Users\Q1\AppData\Local\Continuum\Anaconda2\lib\site-packages\monetdb\sql\connections.py", line 143, in command return self.mapi.cmd(command) File "C:\Users\Q1\AppData\Local\Continuum\Anaconda2\lib\site-packages\monetdb\mapi.py", line 202, in cmd raise OperationalError(response[1:])monetdb.exceptions.OperationalError: 25005!current transaction is aborted (please ROLLBACK) Thanks, --------------------- Nik Patel ________________________________________________________ The information contained in this e-mail is confidential and/or proprietary to Capital One and/or its affiliates and may only be used solely in performance of work or services for Capital One. The information transmitted herewith is intended only for use by the individual or entity to which it is addressed. If the reader of this message is not the intended recipient, you are hereby notified that any review, retransmission, dissemination, distribution, copying or other use of, or taking of any action in reliance upon this information is strictly prohibited. If you have received this communication in error, please contact the sender and delete the material from your computer.

1 0

Announcement: New Jul2015-SP4 Bugfix release of MonetDB suite
by Sjoerd Mullender 11 Apr '16

11 Apr '16

The MonetDB team at CWI/MonetDB BV is pleased to announce the Jul2015-SP4 bugfix release of the MonetDB suite of programs. More information about MonetDB can be found on our website at <http://www.monetdb.org/>. For details on this release, please see the release notes at <http://www.monetdb.org/Downloads/ReleaseNotes>. As usual, the download location is <http://dev.monetdb.org/downloads/>. Jul 2015-SP4 bugfix release MonetDB Common * Fixed a bug that caused various instances where old data returned or where crashes occurred. The problem was that internally data wasn't always marked dirty when it was being changed, causing later processing to not deal with the changed data correctly. Bug Fixes * 3905: MonetDB doesn't handle ANY/SOME/ALL operator correctly * 3929: R aggregate not recognized when using 3 or more parameters * 3965: Not possible to quote/escape single quote character in the name of the file to load. * 3968: Missing double use of column names

1 0

undefined reference to `pcre_config'
by John Krulick 06 Apr '16

06 Apr '16

Hi, I'm trying to build MonetDB from source and running into an error during configure. Running ./configure gives: ... ... checking for SHA256... yes checking for SHA384... yes checking for SHA512... yes checking for pcre... yes checking whether pcre comes with UTF-8 support... no configure: error: PCRE library not found but required for MonetDB5 At the bottom of the email is the error in the config.log. pcretest -C gives me: PCRE version 8.38 2015-11-23 Compiled with:q 8-bit support UTF-8 support Unicode properties support No just-in-time compiler support Newline sequence is LF \R matches all Unicode newlines Internal link size = 2 POSIX malloc threshold = 10 Parentheses nest limit = 250 Default match limit = 10000000 Default recursion depth limit = 10000000 Match recursion uses stack The issue, I think, is I have PCRE version 8.38 built in a non-standard location, /home/john/monetdb_prereq/pcre export pcre_LIBS=-L/home/john/monetdb_prereq/pcre/lib export pcre_CFLAGS=-I/home/john/monetdb_prereq/pcre/include Any idea what I'm missing? Thanks, John configure:20861: checking for SHA512 configure:20861: gcc -o conftest -g -O2 conftest.c -lssl -lcrypto -ldl -lz >&5 configure:20861: $? = 0 configure:20861: result: yes configure:20901: checking for pcre configure:20966: result: yes configure:20971: checking whether pcre comes with UTF-8 support configure:20992: gcc -o conftest -g -O2 -I/home/john/monetdb_prereq/pcre/include conftest.c -L/home/john/monetdb_prereq/pcre/lib >&5 /tmp/ccRjfKcQ.o: In function `main': /home/john/MonetDB-11.21.17/conftest.c:66: undefined reference to `pcre_config' collect2: ld returned 1 exit status configure:20992: $? = 1 configure: program exited with status 1 configure: failed program was: | /* confdefs.h */ | #define PACKAGE_NAME "MonetDB" | #define PACKAGE_TARNAME "MonetDB" | #define PACKAGE_VERSION "11.21.17" | #define PACKAGE_STRING "MonetDB 11.21.17" | #define PACKAGE_BUGREPORT "info(a)monetdb.org" | #define PACKAGE_URL "http://www.monetdb.org/" | #define PACKAGE "MonetDB" | #define VERSION "11.21.17" | #define HOST "x86_64-unknown-linux-gnu" | #define MONETDB_RELEASE "Jul2015-SP3" | #define HAVE_CONSOLE 1 | #define MONETDB5_PASSWDHASH "SHA512" | #define STDC_HEADERS 1 | #define HAVE_SYS_TYPES_H 1 | #define HAVE_SYS_STAT_H 1 | #define HAVE_STDLIB_H 1 | #define HAVE_STRING_H 1 | #define HAVE_MEMORY_H 1 | #define HAVE_STRINGS_H 1 | #define HAVE_INTTYPES_H 1 | #define HAVE_STDINT_H 1 | #define HAVE_UNISTD_H 1 | #define __EXTENSIONS__ 1 | #define _ALL_SOURCE 1 | #define _GNU_SOURCE 1 | #define _POSIX_PTHREAD_SEMANTICS 1 | #define _TANDEM_SOURCE 1 | #define SIZEOF_LONG 8 | #define HAVE_DLFCN_H 1 | #define LT_OBJDIR ".libs/" | #define NDEBUG 1 | #define DIFF "diff" | #define DIR_SEP '/' | #define DIR_SEP_STR "/" | #define PATH_SEP ':' | #define PATH_SEP_STR ":" | #define SO_PREFIX "lib" | #define SO_EXT ".so" | #define HAVE_LIBBZ2 1 | #define HAVE_MALLOPT 1 | #define HAVE_PTHREAD_H 1 | #define HAVE_SEMAPHORE_H 1 | #define HAVE_SCHED_H 1 | #define HAVE_PTHREAD_KILL 1 | #define HAVE_PTHREAD_SIGMASK 1 | #define HAVE_PTHREAD_SETSCHEDPRIO 1 | #define HAVE_LIBPTHREAD 1 | #define HAVE_LIBREADLINE 1 | #define HAVE_ICONV 1 | #define ICONV_CONST | #define HAVE_UUID 1 | #define HAVE_MD5 1 | #define HAVE_RIPEMD160 1 | #define HAVE_SHA1 1 | #define HAVE_SHA224 1 | #define HAVE_SHA256 1 | #define HAVE_SHA384 1 | #define HAVE_SHA512 1 | #define HAVE_OPENSSL 1 | /* end confdefs.h. */ | #include <pcre.h> | int | main () | { | int rc; pcre_config(PCRE_CONFIG_UTF8, &rc); exit(!rc); | ; | return 0; | } configure:20996: result: no configure:21021: error: PCRE library not found but required for MonetDB5

2 2

Announcement: New Jul2015-SP3 Bugfix release of MonetDB suite
by Sjoerd Mullender 25 Mar '16

25 Mar '16

The MonetDB team at CWI/MonetDB BV is pleased to announce the Jul2015-SP3 bugfix release of the MonetDB suite of programs. More information about MonetDB can be found on our website at <http://www.monetdb.org/>. For details on this release, please see the release notes at <http://www.monetdb.org/Downloads/ReleaseNotes>. As usual, the download location is <http://dev.monetdb.org/downloads/>. Jul 2015-SP3 bugfix release MonetDB5 Server * Fixed potential crash in MAL debugger when accessing BATs by index. Functionality dropped as it is also a security leak. Bug Fixes * 2972: SQL URL functionality contains errors * 3549: bulk string operations very slow * 3881: Server crashes on bulk load * 3890: Window function + group by in subselect, rel2bin_project: Assertion `0' failed * 3891: MonetDB crashes when executing SQL with window function * 3900: null handling in some sql statements is incorrect * 3906: Multi-column 1-N table-function with mitosis produces different column counts * 3908: LEFT JOIN with OR conditions triggers assertion * 3909: Incorrect column name in OR condition of LEFT JOIN crashes mserver * 3910: COPY INTO table (column1, column2) got wrong result * 3912: When table/column names conflicts, data ends in multiple tables! * 3917: Date difference returns month_interval instead of day_interval * 3918: MonetDB.R version 1.0.1 incorrectly constructs the batfile script * 3919: Table conflict when the table name and fields are identical * 3921: Creating a table from a complex query crashes mserver or triggers assertion * 3922: AVG( column ) returns NaN rather than Inf when column contains Inf * 3928: When killing a virtual machine, sql_logs/sql/log is empty * 3930: Wrong typecast on character columns in prepared statements when using Umlaute * 3932: CASE expressions with constants are not evaluated correctly * 3933: replace "exit" by "throw new Exception" * 3937: bad BAT properties with binary copy into and NULL values * 3938: Wrong error message on violating foreign key constraint * 3940: Date calculation and comparison produce wrong result * 3941: Wrong coercion priority * 3948: SQL: select * from sys.sys.table_name; is accepted but should return an error * 3951: extern table_funcs not visible from Windows DLL for extensions like vaults (crashes) * 3952: Stream table gives segfault * 3953: MIN/MAX of a UUID column produces wrong results * 3954: Consolidate table assertion error * 3955: (incorrect) MAL loop instead of manifold triggered by simple change in target list

1 0

histogram
by Roberto Cornacchia 21 Mar '16

21 Mar '16

In a C UDF, looping on a [:oid,:str:] bat, I'm tokenizing each string tail into a str array: BATloop(..) { str *token_array = /* create a str array by tokenizing the str tail of this BUN */ /* append a histogram of token_array to result */ } The token array is expected to be in the range of 10 - 10K short strings (1 to 10 bytes). Question: To get the histogram, I'd turn the token_array into a BAT b, then use BATgroup: BATgroup(&gn, NULL, &hn, b, NULL, NULL, NULL); Do you see a more efficient way? Is there a group/histogram primitive implemented that works directly on arrays rather than BATs? Thanks, Roberto

1 0

Error after compiling monetdb
by Shmagi Kavtaradze 19 Mar '16

19 Mar '16

I am trying to write a C User Defined Function. It works in Neatbeans. What I want is, to take value from cells, insert them in text or char array and compare each corresponding array position to each other. As values I will have "1"s and "0s". E.g "101010100011". THe code I wrote in udf.c: /* Comparing bag of words */ char BitAnd(char Cell1[], char Cell2[]) { Cell1[10000]; Cell2[10000]; int counter; float result; int strlen = sizeof(Cell1)/sizeof(Cell1[0])-1; int i; for (i=0; i < strlen; i++) { if (Cell1[i] == Cell2[i]) { counter++; } } result = (float)counter / strlen; } /* MAL wrapper */ char UDFBitAnd(char Cell1[], char Cell2[]) { /* assert calling sanity */ assert(Cell1[] != NULL && Cell2[] != NULL); return MAL_SUCCEED; } In udf.mal I have: command BitAnd(Cell1:chr,Cell2:chr):flt address UDFBitAnd comment "Compare bag of words"; In 80_udf.sql: -- Comparing bag of words create function BitAnd(Cell1[] char, Cell2[] char) returns float external name udf.BitAnd; I compiled Monetdb, but when I try to connect with "monetdb" to db it gives me error. In the log file I found: MSG voc[18857]: !SyntaxException:parseError:command BitAnd(Cell1:chr,Cell2:chr):flt MSG voc[18857]: !SyntaxException:parseError: ^Type identifier expected MSG voc[18857]: !SyntaxException:parseError:command BitAnd(Cell1:chr,Cell2:chr):flt MSG voc[18857]: !SyntaxException:parseError: ^',' expected MSG merovingian[8780]: database 'voc' (18857) has exited with exit status 0 MSG merovingian[8780]: database 'voc' has shut down ERR merovingian[8780]: client error: database 'voc' started up, but failed to open up a communication channel

1 0

Adapting BATs
by Till Kolditz 18 Mar '16

18 Mar '16

Hi there, I'm just new to the list, so hello to everyone :-) I'm currently trying to find out how to adapt BATs in MonetDB for a scientific project. What I want to do is to add an encoding to data as some kind of a native DB feature. I.e. that a BAT stores data in some encoded fashion to detect certain kinds of errors that alter data in memory. By that, talking e.g. about an int BAT (column), the data width would increase e.g. from 4 to 5 Bytes (just as an example) where the additional byte stores the coding information. As I see it there are for a first shot two possibilities to do that: 1) Take one atom (e.g. int) and adapt all BAT / atom access code which touches int BATs. 2) Add some kind of new atom which is handled in a specific way. Since I have just started looking into MonetDB I now hope for a few hints to decide whether one of these approaches is sensible and whether there is maybe a simpler way. Thanks a lot in advance for any helpful comments :-) Best Till Kolditz -- Dipl.-Inf. Till Kolditz wiss. Mitarbeiter Technische Universität Dresden Fakultät Informatik Lehrstuhl Datenbanken 01062 Dresden Tel.: +49 (351) 463-38077 Fax: +49 (351) 463-38259 E-Mail: Till.Kolditz(a)tu-dresden.de jExam Group www: http://www.jexam.de Mail: till.kolditz(a)jexam.de

1 0

New Jun2016 stable branch created
by Sjoerd Mullender 14 Mar '16

14 Mar '16

I have created a new stable branch for the next feature release. This branch was created in order to stabilize the code before the feature release. From now on *only* bug fixes may go to this new stable branch. I reserve the right to undo any changes that do not comply with this rule. I also reserve the right to admit changes that are not strictly bug fixes. The name for the branch is Jun2016. So in each of your stable checkout directories you can do hg pull hg update -rJun2016 to update to this new branch. -- Sjoerd Mullender

1 0

Re: MonetDB: leftmart - trying to fix Windows compilation: do not "i...
by Lefteris 08 Mar '16

08 Mar '16

Stefan, this needs to be inlined because it is used many times. I will deal with compilation issues later on. This is a dev branch and we are running different experiments every night. But with this change if measured times are different I would not know if it is because of other changes or because of the inline statement removal. I would appreciate if you did not make any changes in this branch at the moment. thank you, lefteris On Mon, Mar 7, 2016 at 8:34 PM, Stefan Manegold < commits+Stefan.Manegold=cwi.nl(a)monetdb.org> wrote: > Changeset: ea1aed8beff8 for MonetDB > URL: http://dev.monetdb.org/hg/MonetDB?cmd=changeset;node=ea1aed8beff8 > Modified Files: > gdk/gdk_bloom.c > Branch: leftmart > Log Message: > > trying to fix Windows compilation: do not "inline" BLOOMask() > > On Windows, compilation fails with > failed invocation: cl -GF -W3 -WX -MD -nologo -Ox -Zi -Oi -I. -I.\.. > -I.\..\..\gdk -I..\common\options -I.\..\..\gdk\..\common\options > -I..\common\stream -I.\..\..\gdk\..\common\stream -I..\common\utils > -I.\..\..\gdk\..\common\utils -I.\..\..\buildtools\conf -DHAVE_ICONV > -IC:\Libraries\iconv-1.11.1.win64-vs2010\include -DHAVE_LIBATOMIC_OPS > -IC:\Libraries\atomic_ops-7.4.2 -DNDEBUG -LD -Felibbat.dll gdk_batop.obj > gdk_select.obj gdk_search.obj gdk_hash.obj gdk_tm.obj gdk_orderidx.obj > gdk_align.obj gdk_bbp.obj gdk_heap.obj gdk_utils.obj gdk_atoms.obj > gdk_qsort.obj gdk_ssort.obj gdk_storage.obj gdk_bat.obj gdk_delta.obj > gdk_cross.obj gdk_system.obj gdk_value.obj gdk_posix.obj gdk_logger.obj > gdk_sample.obj gdk_calc.obj gdk_aggr.obj gdk_group.obj gdk_imprints.obj > gdk_bloom.obj gdk_join.obj gdk_project.obj gdk_unique.obj gdk_firstn.obj > libbat.res /link ..\common\options\monet_options.obj > ..\common\stream\libstream.lib ..\common\utils\mutils.obj > ..\common\utils\prompt.obj wsock32.lib Ws2_32.l > ib C:\Libraries\zlib-1.2.8.win64-vs2010\lib\zdll.lib > C:\Libraries\bzip2-1.0.6.win64-vs2010\lib\libbz2.lib psapi.lib > Creating library libbat.lib and object libbat.exp > gdk_join.obj : error LNK2019: unresolved external symbol BLOOMask > referenced in function hashjoin > libbat.dll : fatal error LNK1120: 1 unresolved externals > > > Maybe, the fact that BLOOMask() is "inline"d triggers this problem ... > > > diffs (11 lines): > > diff --git a/gdk/gdk_bloom.c b/gdk/gdk_bloom.c > --- a/gdk/gdk_bloom.c > +++ b/gdk/gdk_bloom.c > @@ -240,7 +240,6 @@ BATbloom(BAT *b) > return GDK_SUCCEED; > } > > -inline > int BLOOMask(BUN v, Bloomfilter *bloom) > { > BUN hv,mv,x,y,z; > _______________________________________________ > checkin-list mailing list > checkin-list(a)monetdb.org > https://www.monetdb.org/mailman/listinfo/checkin-list >

3 2

Re: MonetDB: leftmart - silence some compilers compaining about prin...
by Stefan Manegold 07 Mar '16

07 Mar '16

Hi Lefteris, the portable format string that we ussually use for size_t in MonetDB is "SZFMT" Best, Stefan ----- On Mar 7, 2016, at 2:38 PM, lsidir commits+lsidir=cwi.nl(a)monetdb.org wrote: > Changeset: a76605fb0b33 for MonetDB > URL: http://dev.monetdb.org/hg/MonetDB?cmd=changeset;node=a76605fb0b33 > Modified Files: > gdk/gdk_bloom.c > Branch: leftmart > Log Message: > > silence some compilers compaining about printing a size_t as unsigned int or > long > > > diffs (12 lines): > > diff --git a/gdk/gdk_bloom.c b/gdk/gdk_bloom.c > --- a/gdk/gdk_bloom.c > +++ b/gdk/gdk_bloom.c > @@ -183,7 +183,7 @@ BATbloom(BAT *b) > > ALGODEBUG fprintf(stderr, "#BATbloom(b=%s#" BUNFMT ") %s: " > "create bloom filter: mbits = " BUNFMT ", ratio = " BUNFMT > - ", kfunc = %d, bytes = " BUNFMT "\n", BATgetId(b), > + ", kfunc = %d, bytes = %lu\n", BATgetId(b), > BATcount(b), b->T->heap.filename, > bloom->mbits, bloom->mbits/BATcount(b), > bloom->kfunc, bloom->bytes); > _______________________________________________ > checkin-list mailing list > checkin-list(a)monetdb.org > https://www.monetdb.org/mailman/listinfo/checkin-list -- | Stefan.Manegold(a)CWI.nl | DB Architectures (DA) | | www.CWI.nl/~manegold/ | Science Park 123 (L321) | | +31 (0)20 592-4212 | 1098 XG Amsterdam (NL) |

1 0

Monetdb database crashes
by Souri Ratnaparkhi 04 Mar '16

04 Mar '16

Hello Team, Hello monetdb team, We have a table with 126 columns and approx. 94000 rows. We are trying to load approx 2500 records from temporary table into main table using query “insert into select” query. The database crashes upon query execution. No specific error is created in log file. When tried with 1500 records, it works. We also created another table with same DDL, that table is empty & inserted all the records from temp successfully without any issue. What could be the reason? Is 126 cols an issue? Please help. We are stuck. (btw: We have machine about 200GB of hard disk & 16 GB RAM (14 GB free)) Thank you [image: Description: Description: Description: Description: logo] * Souri Ratnaparkhi United Software Associates Pvt. Ltd. Pune, India Mob: 8975677255* * Phone (India Lines): (91) 020-25445550 **www.usain.com* <http://www.usain.com/>

2 1

Error delete rows on table
by Luciano Sasso 24 Feb '16

24 Feb '16

Hi, I have two tables , deleted records of the two tables , then tried to delete again in Table 1 , an error occurred ( could not get print) , and in Table 2 records erased, again show in Table 2. Everything was running on a SQL console ( NetBeans ) and auto commit on. -- Luciano Sasso Vieira Data Scientist & Solutions Architect luciano(a)gsgroup.com.br <http://www.gsgroup.com.br> | tel: 17 3353-0833 | cel: 17 99706-9335 www.gsgroup.com.br <http://www.gsgroup.com.br> --- Este email foi escaneado pelo Avast antivírus. https://www.avast.com/antivirus

1 0

Slow query
by Luciano Sasso 24 Feb '16

24 Feb '16

Hi, I have MonetDB JUL2015-SP2, with table of 100M rows and 20 coluns, 4 of number type, others string type. A simple "select count(*) from table", run on 90 minutes. Hardware and SO is: Debian 8, 4 cores, 11G RAM and 20G free on disc. While monet run the query, command top, show 42G of virtual memory. Its normal ?, any idea ? thanks. -- Luciano Sasso Vieira Data Scientist & Solutions Architect luciano(a)gsgroup.com.br <http://www.gsgroup.com.br> | tel: 17 3353-0833 | cel: 17 99706-9335 www.gsgroup.com.br <http://www.gsgroup.com.br> --- Este email foi escaneado pelo Avast antivírus. https://www.avast.com/antivirus

1 0

Re: MonetDB: default - use same optimizer pipe for inlined queries.
by Stefan Manegold 03 Feb '16

03 Feb '16

cwi@achtbaan:~/manegold/source$ hg clone http://dev.monetdb.org/hg/MonetDB/ Het programma ‘hg’ is momenteel niet geïnstalleerd. Vraag uw beheerder om het pakket 'mercurial' te installeren om 'hg' uit te kunnen voeren nou, dan maar wget de tarball - werkt maar: cwi@achtbaan:~/manegold/build$ ../source/MonetDB-11.21.13/configure --enable-optimize --prefix=/home/cwi/manegold/prefix 2>&1 | tee config.out checking build system type... powerpc64le-unknown-linux-gnu checking host system type... powerpc64le-unknown-linux-gnu checking target system type... powerpc64le-unknown-linux-gnu checking for a BSD-compatible install... /usr/bin/install -c checking whether build environment is sane... yes checking for a thread-safe mkdir -p... /bin/mkdir -p checking for gawk... no checking for mawk... mawk checking whether make sets $(MAKE)... yes checking whether make supports nested variables... yes checking whether UID '1001' is supported by ustar format... yes checking whether GID '1001' is supported by ustar format... yes checking how to create a ustar tar archive... gnutar checking for pkg-config... no configure: error: pkg-config is required to configure MonetDB ... jammer ... Stefan ----- On Feb 3, 2016, at 10:45 AM, Niels Nes commits+niels=cwi.nl(a)monetdb.org wrote: > Changeset: 039a2b238d13 for MonetDB > URL: http://dev.monetdb.org/hg/MonetDB?cmd=changeset;node=039a2b238d13 > Modified Files: > sql/backends/monet5/sql_optimizer.c > sql/backends/monet5/sql_optimizer.h > sql/backends/monet5/sql_scenario.c > Branch: default > Log Message: > > use same optimizer pipe for inlined queries. > > > diffs (107 lines): > > diff --git a/sql/backends/monet5/sql_optimizer.c > b/sql/backends/monet5/sql_optimizer.c > --- a/sql/backends/monet5/sql_optimizer.c > +++ b/sql/backends/monet5/sql_optimizer.c > @@ -215,8 +215,8 @@ addOptimizers(Client c, MalBlkPtr mb, ch > addtoMalBlkHistory(mb, "getStatistics"); > } > > -void > -addQueryToCache(Client c) > +str > +optimizeQuery(Client c) > { > MalBlkPtr mb; > backend *be; > @@ -226,13 +226,12 @@ addQueryToCache(Client c) > assert(be && be->mvc); /* SQL clients should always have their state set */ > pipe = getSQLoptimizer(be->mvc); > > - insertSymbol(c->nspace, c->curprg); > trimMalBlk(c->curprg->def); > c->blkmode = 0; > mb = c->curprg->def; > chkProgram(c->fdout, c->nspace, mb); > #ifdef _SQL_OPTIMIZER_DEBUG > - mnstr_printf(GDKout, "ADD QUERY TO CACHE\n"); > + mnstr_printf(GDKout, "Optimize query\n"); > printFunction(GDKout, mb, 0, LIST_MAL_ALL); > #endif > /* > @@ -250,23 +249,34 @@ addQueryToCache(Client c) > if (msg != MAL_SUCCEED) > GDKfree(msg); /* ignore error */ > } > - return; > + return NULL; > } > addOptimizers(c, mb, pipe); > msg = optimizeMALBlock(c, mb); > - if (msg != MAL_SUCCEED) { > - showScriptException(c->fdout, mb, 0, MAL, "%s", msg); > - GDKfree(msg); > - return; > - } > + if (msg) > + return msg; > > /* time to execute the optimizers */ > if (c->debug) > optimizerCheck(c, mb, "sql.baseline", -1, 0); > #ifdef _SQL_OPTIMIZER_DEBUG > - mnstr_printf(GDKout, "ADD optimized QUERY TO CACHE\n"); > + mnstr_printf(GDKout, "End Optimize Query\n"); > printFunction(GDKout, mb, 0, LIST_MAL_ALL); > #endif > + return NULL; > +} > + > +void > +addQueryToCache(Client c) > +{ > + str msg = NULL; > + > + insertSymbol(c->nspace, c->curprg); > + msg = optimizeQuery(c); > + if (msg != MAL_SUCCEED) { > + showScriptException(c->fdout, c->curprg->def, 0, MAL, "%s", msg); > + GDKfree(msg); > + } > } > > /* > diff --git a/sql/backends/monet5/sql_optimizer.h > b/sql/backends/monet5/sql_optimizer.h > --- a/sql/backends/monet5/sql_optimizer.h > +++ b/sql/backends/monet5/sql_optimizer.h > @@ -12,6 +12,7 @@ > > //#define _SQL_OPTIMIZER_DEBUG > > +sql5_export str optimizeQuery(Client c); > sql5_export void addQueryToCache(Client c); > sql5_export str SQLoptimizer(Client c); > sql5_export void SQLsetAccessMode(Client c); > diff --git a/sql/backends/monet5/sql_scenario.c > b/sql/backends/monet5/sql_scenario.c > --- a/sql/backends/monet5/sql_scenario.c > +++ b/sql/backends/monet5/sql_scenario.c > @@ -1263,17 +1263,20 @@ recompilequery: > > chkTypes(c->fdout, c->nspace, c->curprg->def, TRUE); /* resolve types */ > if (opt) { > + str msg = optimizeQuery(c); > + /* > MalBlkPtr mb = c->curprg->def; > > trimMalBlk(mb); > chkProgram(c->fdout, c->nspace, mb); > addOptimizers(c, mb, "default_pipe"); // TODO change to active pipe! > msg = optimizeMALBlock(c, mb); > + */ > if (msg != MAL_SUCCEED) { > sqlcleanup(m, err); > goto finalize; > } > - c->curprg->def = mb; > + //c->curprg->def = mb; > } > //printFunction(c->fdout, c->curprg->def, 0, LIST_MAL_ALL); > /* we know more in this case than chkProgram(c->fdout, c->nspace, > c->curprg->def); */ > _______________________________________________ > checkin-list mailing list > checkin-list(a)monetdb.org > https://www.monetdb.org/mailman/listinfo/checkin-list -- | Stefan.Manegold(a)CWI.nl | DB Architectures (DA) | | www.CWI.nl/~manegold/ | Science Park 123 (L321) | | +31 (0)20 592-4212 | 1098 XG Amsterdam (NL) |

1 0

Drastic speed improvement for str.toLower() / str.toUpper
by Roberto Cornacchia 18 Jan '16

18 Jan '16

Hello, I had reported in https://www.monetdb.org/bugzilla/show_bug.cgi?id=3549 that string case conversion is very inefficient in MonetDB. I had a look at the code. For each UTF8 character it performs a hash lookup from the origin case bat and finds the corresponding character in the destination case bat. However, this is an overkill for ASCII characters: - for letters, [A-Z] + 32 = [a-z] - all other ASCII characters stay the same With the assumption that single-byte characters are very frequent in most texts, it makes sense to invest in a simple test and perform the hash lookup only for multi-byte characters. I tested this on 831MB (over 360K tuples) of standard English text: - original str.toLower/str.toUpper: 101 seconds (8 MB/s) - modified version: 3.6 seconds (230 MB/s) I guess that even when the text is highly multi-byte oriented the added test wouldn't hurt that much. A side-observation perhaps worth investigating is why that hash lookup is so expensive. Please find my patch in attachment. Roberto

2 3

Announcement: New Jul2015-SP2 Bugfix release of MonetDB suite
by Sjoerd Mullender 15 Jan '16

15 Jan '16

The MonetDB team at CWI/MonetDB BV is pleased to announce the Jul2015-SP2 bugfix release of the MonetDB suite of programs. More information about MonetDB can be found on our website at <http://www.monetdb.org/>. For details on this release, please see the release notes at <http://www.monetdb.org/Downloads/ReleaseNotes>. As usual, the download location is <http://dev.monetdb.org/downloads/>. Jul 2015-SP2 bugfix release Bug Fixes * 2014: 'null' from copy into gets wrong * 3817: opt_pushselect stuck with multi-table UDF * 3835: windows does not release ram after operations * 3836: rand() only gets evaluated once when used as an expression * 3838: Update column with or without parenthesis produce different results * 3840: savepoints may crash the database * 3841: mclient fails with response "Challenge string is not valid" * 3842: SQL execution fails to finish and reports bogus error messages * 3845: Too many VALUES in INSERT freeze mserver5 * 3847: Wrong SQL results for a certain combination of GROUP BY / ORDER BY / LIMIT * 3848: mserver segfault during bulk loading/updating * 3849: HUGEINT incorrect value * 3850: DEL character not escaped * 3851: expression that should evaluate to FALSE evaluates to TRUE in SELECT query * 3852: CASE statement produces GDK error on multithreaded database: BATproject does not match always * 3854: Complex expression with comparison evaluates incorrectly in WHERE clause * 3855: Foreign key referencing table in a different schema - not allowed. * 3857: Large LIMIT in SELECT may abort the query * 3861: Using window functions cause a crash * 3864: Error in bulk import for chinese character * 3871: NOT x LIKE triggers "too many nested operators" * 3872: mserver crashes under specific combination of JOIN and WHERE conditions * 3873: mserver5: gdk_bat.c:1015: setcolprops: Assertion `x != ((void *)0) || col->type == 0' failed. * 3879: Database crashes when querying with several UNION ALLs. * 3887: Querying "sys"."tracelog" causes assertion violation and crash of mserver5 process * 3889: read only does not protect empty tables * 3895: read only does not protect this table

1 0

About JDBC Driver
by Luciano Sasso 07 Jan '16

07 Jan '16

Steps for error: 1- In testing with java client and JDBC 2:19, an error occurs when saving decimal field, for example the value 11:23 is recorded as 13. 2- When I run the same test with the 2.8 driver, saves normally and correct. Not only is the 2.19 driver, all above 2.8 is that same mistake. My Java client is Pentaho Data Integration, My MonetDB is JUL2015-SPI I go zip package test, and send fo tests. Att, -- Luciano Sasso Vieira Data Scientist & Solutions Architect luciano(a)gsgroup.com.br <http://www.gsgroup.com.br> | tel: 17 3353-0833 | cel: 17 99706-9335 www.gsgroup.com.br <http://www.gsgroup.com.br> --- Este email foi escaneado pelo Avast antivírus. https://www.avast.com/antivirus

1 0

Target Branch?
by Jörg Strebel 05 Jan '16

05 Jan '16

Hello! Thanks for the answer. What would be the branch to base my patches / developments on? I see two candidates in http://dev.monetdb.org/hg/MonetDB/branches: - default - Jul2015 Could you advise? Thanks! > The documentation for setting up the development environment and for > getting the source code is pretty good, but I am missing a process > description, how a volunteer can participate in the software > development process. E.g. how are bugs assigned, You could start by picking a bug, trying to fix it and then attach the patch to the bug description. We will have a a look at it for sure. > submitted, what needs to be done besides code production? Writing test cases for bugs without them is also a nice way of getting started. -- Jörg Strebel Aachener Straße 2 80804 München

2 1

How to contribute to MonetDB
by Jörg Strebel 04 Jan '16

04 Jan '16

Hello! I find MonetDB as a software product interesting and would be interested in contributing to its development (and learning about In-Memory databases in the process) . The documentation for setting up the development environment and for getting the source code is pretty good, but I am missing a process description, how a volunteer can participate in the software development process. E.g. how are bugs assigned, how are patches submitted, what needs to be done besides code production? Is there a description of sorts? Best regards Jörg -- Jörg Strebel Aachener Straße 2 80804 München

3 2

How is a query with 'string' columns in it, gets executed?
by bhuvan Mitra 04 Jan '16

04 Jan '16

Hello, I am looking into these column oriented databases for a while and wondered with monetdb's performance over others. I started studying about monet and found very interesting. Many concepts you use are really interesting. I want to know how monet executes a query which contain string columns in it. Does it maintain a dictionary or what kind of index it uses or is there any inverted index involved for strings etc., Any kind of material, source code portions or any other documentation regarding this, is most welcome. I would love to contribute back from my side once I get to know what is happening. Regards, Mit

2 1

MonetDB: reserved SQL keywords
by martin van dinther 14 Dec '15

14 Dec '15

Hi Hannes, FYI, we have a system table sys.keywords with all the reserved SQL keywords. The intention of this table is to have one place (in the server) with all the reserved SQL keywords, such that API implementations (ODBC, JDBC, PHP, Ruby, Python, R, etc) can query that table and construct the list or string. In the default branch I have implemented this already for jdbc/MonetDatabaseMetaData.java method getSQLKeywords(). This should reduce future maintenance of all the places where these reserved SQL keywords are hardcoded. The new keywords can simply be added/ in one place, the keywords table. It is filled in sql/scripts/51_sys_schema_extension.sql Regards, Martin van Dinther On 24-11-15 09:42, Hannes Muehleisen wrote: > Changeset: c0fb9a0d91e3 for MonetDB > URL: http://dev.monetdb.org/hg/MonetDB?cmd=changeset;node=c0fb9a0d91e3 > Modified Files: > clients/R/MonetDB.R/R/dbi.R > Branch: embedded > Log Message: > > R Connector: More naughty words > > > diffs (65 lines): > > diff --git a/clients/R/MonetDB.R/R/dbi.R b/clients/R/MonetDB.R/R/dbi.R > --- a/clients/R/MonetDB.R/R/dbi.R > +++ b/clients/R/MonetDB.R/R/dbi.R > @@ -391,24 +391,43 @@ setMethod("dbSendQuery", signature(conn= > invisible(new("MonetDBEmbeddedResult", env=env)) > }) > > - > -reserved_monetdb_keywords <- c(.SQL92Keywords, > -"ADMIN", "AFTER", "AGGREGATE", "ALWAYS", "ASYMMETRIC", "ATOMIC", > -"AUTO_INCREMENT", "BEFORE", "BEST", "BIGINT", "BIGSERIAL", "BINARY", > -"BLOB", "CALL", "CHAIN", "CLOB", "COMMITTED", "COPY", "CROSS", > -"CURRENT_ROLE", "CURRENT_TIME", "CURRENT_USER", "DELIMITERS", > -"DO", "EACH", "EFFORT", "ELSEIF", "ENCRYPTED", "EXCLUDE", "FOLLOWING", > -"FUNCTION", "GENERATED", "HUGEINT", "IF", "ILIKE", "LIMIT", "LOCALTIME", > -"LOCALTIMESTAMP", "LOCKED", "MERGE", "NATURAL", "NEW", "NOCYCLE", > -"NOMAXVALUE", "NOMINVALUE", "OFFSET", "OLD", "ON", "OTHERS", > -"OVER", "PARTITION", "PRECEDING", "RANGE", "RECORDS", "REFERENCING", > -"REMOTE", "RENAME", "REPEATABLE", "REPLICA", "RESTART", "RETURN", > -"RETURNS", "SAMPLE", "SAVEPOINT", "SEQUENCE", "SERIAL", "SERIALIZABLE", > -"SESSION_USER", "SIMPLE", "SPLIT_PART", "STDIN", "STDOUT", "STREAM", > -"SYMMETRIC", "TIES", "TINYINT", "TRIGGER", "UNBOUNDED", "UNCOMMITTED", > -"UNENCRYPTED", "WHILE", "XMLAGG", "XMLATTRIBUTES", "XMLCOMMENT", > -"XMLCONCAT", "XMLDOCUMENT", "XMLELEMENT", "XMLFOREST", "XMLNAMESPACES", > -"XMLPARSE", "XMLPI", "XMLQUERY", "XMLSCHEMA", "XMLTEXT", "XMLVALIDATE") > +# found from sql_scan.c and trial/error > +reserved_monetdb_keywords <- sort(unique(toupper(c(.SQL92Keywords, > +"ADD", "ADMIN", "AFTER", "AGGREGATE", "ALL", "ALTER", "ALWAYS", > +"AND", "ANY", "ASC", "ASYMMETRIC", "ATOMIC", "AUTO_INCREMENT", > +"BEFORE", "BEGIN", "BEST", "BETWEEN", "BIGINT", "BIGSERIAL", > +"BINARY", "BLOB", "BY", "CALL", "CASCADE", "CASE", "CAST", "CHAIN", > +"CHAR", "CHARACTER", "CHECK", "CLOB", "COALESCE", "COMMIT", "COMMITTED", > +"CONSTRAINT", "CONVERT", "COPY", "CORRESPONDING", "CREATE", "CROSS", > +"CURRENT", "CURRENT_DATE", "CURRENT_ROLE", "CURRENT_TIME", "CURRENT_TIMESTAMP", > +"CURRENT_USER", "DAY", "DEC", "DECIMAL", "DECLARE", "DEFAULT", > +"DELETE", "DELIMITERS", "DESC", "DO", "DOUBLE", "DROP", "EACH", > +"EFFORT", "ELSE", "ELSEIF", "ENCRYPTED", "END", "ESCAPE", "EVERY", > +"EXCEPT", "EXCLUDE", "EXISTS", "EXTERNAL", "EXTRACT", "FALSE", > +"FLOAT", "FOLLOWING", "FOR", "FOREIGN", "FROM", "FULL", "FUNCTION", > +"GENERATED", "GLOBAL", "GRANT", "GROUP", "HAVING", "HOUR", "HUGEINT", > +"IDENTITY", "IF", "ILIKE", "IN", "INDEX", "INNER", "INSERT", > +"INT", "INTEGER", "INTERSECT", "INTO", "IS", "ISOLATION", "JOIN", > +"LEFT", "LIKE", "LIMIT", "LOCAL", "LOCALTIME", "LOCALTIMESTAMP", > +"LOCKED", "MEDIUMINT", "MERGE", "MINUTE", "MONTH", "NATURAL", > +"NEW", "NEXT", "NOCYCLE", "NOMAXVALUE", "NOMINVALUE", "NOT", > +"NOW", "NULL", "NULLIF", "NUMERIC", "OF", "OFFSET", "OLD", "ON", > +"ONLY", "OPTION", "OR", "ORDER", "OTHERS", "OUTER", "OVER", "PARTIAL", > +"PARTITION", "POSITION", "PRECEDING", "PRESERVE", "PRIMARY", > +"PRIVILEGES", "PROCEDURE", "PUBLIC", "RANGE", "READ", "REAL", > +"RECORDS", "REFERENCES", "REFERENCING", "REMOTE", "RENAME", "REPEATABLE", > +"REPLICA", "RESTART", "RESTRICT", "RETURN", "RETURNS", "REVOKE", > +"RIGHT", "ROLLBACK", "ROWS", "SAMPLE", "SAVEPOINT", "SECOND", > +"SELECT", "SEQUENCE", "SERIAL", "SERIALIZABLE", "SESSION_USER", > +"SET", "SIMPLE", "SMALLINT", "SOME", "SPLIT_PART", "STDIN", "STDOUT", > +"STORAGE", "STREAM", "STRING", "SUBSTRING", "SYMMETRIC", "THEN", > +"TIES", "TINYINT", "TO", "TRANSACTION", "TRIGGER", "TRUE", "UNBOUNDED", > +"UNCOMMITTED", "UNENCRYPTED", "UNION", "UNIQUE", "UPDATE", "USER", > +"USING", "VALUES", "VARCHAR", "VARYING", "VIEW", "WHEN", "WHERE", > +"WHILE", "WITH", "WORK", "WRITE", "XMLAGG", "XMLATTRIBUTES", > +"XMLCOMMENT", "XMLCONCAT", "XMLDOCUMENT", "XMLELEMENT", "XMLFOREST", > +"XMLNAMESPACES", "XMLPARSE", "XMLPI", "XMLQUERY", "XMLSCHEMA", > +"XMLTEXT", "XMLVALIDATE", "YEAR")))) > > # quoting > quoteIfNeeded <- function(conn, x, warn=T, ...) { > _______________________________________________ > checkin-list mailing list > checkin-list(a)monetdb.org > https://www.monetdb.org/mailman/listinfo/checkin-list

2 1