- users-list - monetdb.org

(no subject)
by musa 10 Jul '15

10 Jul '15

Can someone please provide a minimal example (how to start the server) for running the embedded R feature in windows? Regards, mpa

3 2

RE: Concurrency test causes crash of MonetDB 5 server on Windows
by michael.witt＠uniserv.com 08 Jul '15

08 Jul '15

Hello, Guillaume: Thanks for the tip avoiding concurrency conflicts. We already considered that and changed to code to avoid such conflicts. In the meanwhile I tested both on Windows and Linux machines and now it is clear that it is a problem specific to the Windows builds of MonetDB. The mserver5 on Windows starts with 2 threads serving requests as stated in the console after starting... # Serving database 'uniserv', using 2 threads Then I found a statement about the thread implementation … [View More]on Windows by Stefan Mangegold: https://www.monetdb.org/pipermail/developers-list/2010-July/003492.html "However, we do not know how well the multi-threading works on Windows. We rely on pthreads, and their implementation on Windows might not be as efficient and/or stable as on Unix-like systems ..." This is absolutely true! My tests show that mserver5 started with 2 threads always crashes after some time firing concurrent read requests. By the way, the windows machine has 2 physical and 2 logical cores. But if I do the same on a Linux machine (also with 2 threads) Mserver5 runs stable. What is the solution to stabilize Mserver5 on Windows? Set the number of threads to 1 by setting the start option: --set gdk_nr_threads=1 In the M5server.bat script. After setting that, Mserver5 run stable on Windows, too. Does anyone know better solutions? Kind regards, Michael -----Original Message----- From: users-list [mailto:users-list-bounces+michael.witt=uniserv.com@monetdb.org] On Behalf Of Guillaume Bassot Sent: Dienstag, 30. Juni 2015 10:26 To: Communication channel for MonetDB users Subject: Re: Concurrency test causes crash of MonetDB 5 server Hello Michael, From my experience, all "write" request executed concurrently always end with concurrency conflict. I'm talking of heavey request during several minutes, so it's easy to get concurrent access. Since i only use monetdb with a single process, stability was greatly improved. Regards Guillaume BASSOT Web Developer +33 (0)1 55 48 02 82 gbassot(a)inbox.fr <mailto:gbassot@inbox.fr> 30, rue Vincent Moris 92240 Malakoff - France Fax : +33 (0)1 55 48 02 79 2015-06-30 10:09 GMT+02:00 <michael.witt(a)uniserv.com <mailto:michael.witt@uniserv.com> >: Hi all, It’s a pity that no one answered the question I asked a few weeks ago about crashes of MonetDB 5 server under windows 64 Bit. So I think I offer more information about a scenario where the databases crashes. Hopefully someone could give me a hint how to improve stability of MonetDB under Windows 64 Bit with concurrent client connections. We are using Monet DB for a couple of months and facing some graved issues with concurrent queries. The problem is, when we query the database with two or more clients in a parallel, the database crashes! These are the facts: MonetDB server version: v11.19.9 ================================ # MonetDB 5 server v11.19.9 "Oct2014-SP2" # Serving database 'uniserv', using 2 threads # Compiled for x86_64-pc-winnt/64bit with 64bit OIDs dynamically linked # Found 8.000 GiB available main-memory. # Copyright (c) 1993-July 2008 CWI. # Copyright (c) August 2008-2015 MonetDB B.V., all rights reserved # Visit http://www.monetdb.org/ <http://www.monetdb.org/> for further information # Listening for connection requests on mapi:monetdb://QADQSHWIN:50000/ # MonetDB/SQL module loaded Server crashes also occur on versions v11.19.8 and v11.19.7. Operation system: ================= Microsoft Windows Server 2008 R2 Datacenter Version 6.1.7601 Service Pack 1 Build 7601 hosted on VMWare Virtual Platform x64-based PC 2x Processor Intel(R) Xeon(R) CPU E5-2680 v2 @ 2.80GHz, 2800 Mhz, 1 Core(s), 1 Logical Processor(s) Installed Physical Memory (RAM) 8,00 GB Available Physical Memory 5,25 GB Total Virtual Memory 16,0 GB Server installation folder: =========================== E:\Program Files\MonetDB\MonetDB5 Server startup script M5server.bat: =================================== @REM Copyright August 2008-2014 MonetDB B.V. @REM All Rights Reserved. @echo off setlocal rem figure out the folder name set MONETDB=%~dp0 rem remove the final backslash from the path set MONETDB=%MONETDB:~0,-1% rem extend the search path with our EXE and DLL folders set PATH=%MONETDB%\bin;%MONETDB%\lib;%MONETDB%\lib\MonetDB5;%PATH% rem prepare the arguments to mserver5 to tell it where to put the dbfarm REM Uniserv: Don't use APPDATA, so set APPDATA to empty String. set APPDATA= if "%APPDATA%" == "" goto usevar rem if the APPDATA variable does exist, put the database there set MONETDBDIR=%APPDATA%\MonetDB5 set MONETDBFARM="--dbpath=%MONETDBDIR%\dbfarm\demo" goto skipusevar :usevar rem if the APPDATA variable does not exist, put the database in the rem installation folder (i.e. default location, so no command line argument) REM Uniserv: Specify the DB installation folder rem set MONETDBDIR=%MONETDB%\var\MonetDB5 set MONETDBDIR=%MONETDB%\db set MONETDBFARM="--dbpath=%MONETDBDIR%\dbfarm\uniserv" :skipusevar rem the SQL log directory used to be in %MONETDBDIR%, but we now rem prefer it inside the dbfarm, so move it there if not exist "%MONETDBDIR%\sql_logs" goto skipmove for /d %%i in ("%MONETDBDIR%"\sql_logs\*) do move "%%i" "%MONETDBDIR%\dbfarm"\%%~ni\sql_logs rmdir "%MONETDBDIR%\sql_logs" :skipmove rem start the real server "%MONETDB%\bin\mserver5.exe" --set "prefix=%MONETDB%" --set mapi_open=true --set mapi_port=50000 --set max_clients=1000 --set "exec_prefix=%MONETDB%" %MONETDBFARM% %* if ERRORLEVEL 1 pause endlocal MonetDB JDBC Client: ================ We query the database via the latest MonetDB JDBC driver loaded from the central Maven repository: <groupId>monetdb</groupId> <artifactId>monetdb-jdbc</artifactId> <version>2.14</version> Database tables: ================ We create tables called "rule_result_<n>" CREATE TABLE "dq_scorecard"."rule_results_1" ("id" BIGINT NOT NULL PRIMARY KEY AUTO_INCREMENT, "record_id" VARCHAR(1024), "rule_1" INT, "rule_2" INT, ...); Each table has about 50+ INT columns. Query scenarios: ================ We have concurrent read access via JDBC driver and Spring Data Access Objects, the query look like that: *Client 1: 50 aggregate queries on 50 different columns on one database table like that: SELECT AVG("rule_n") FROM "dq_scorecard"."rule_results_1" WHERE "rule_n" >= 0 *Client 2: 50 aggregate queries on 50 different columns on another database table like that: SELECT AVG("rule_n") FROM "dq_scorecard"."rule_results_2" WHERE "rule_n" >= 0 *Client 3: 50 aggregate queries on 50 different columns on another database table like that: SELECT AVG("rule_n") FROM "dq_scorecard"."rule_results_3" WHERE "rule_n" >= 0 All clients run concurrently, at the moment we don't use a connection pool and create a connection for every query. During concurrent access the M5server crashes. The server crashes when each table has 100 records! Our production tables can have up to 50. 000. 000 of records! Kind regards, Michael From: users-list [mailto:users-list-bounces+michael.witt <mailto:users-list-bounces%2Bmichael.witt> =uniserv.com(a)monetdb.org <mailto:uniserv.com@monetdb.org> ] On Behalf Of michael.witt(a)uniserv.com <mailto:michael.witt@uniserv.com> Sent: Donnerstag, 28. Mai 2015 14:32 To: users-list(a)monetdb.org <mailto:users-list@monetdb.org> Subject: Concurrency test causes crash of MonetDB 5 server Hi, we experience a crash of MonetDB 5 server during tests of a multi-threaded client access via JDBC-Driver and MapiSocket: The application is already optimized to avoid “concurrency conflicts” because of the “Optimistic Concurrency Control” concept. The test simulates client access via two threads. All update operations (INSERT, UPDATE, DELETE, DROP TABLE, CREATE TABLE) are done in serial. All read operations are done in parallel to other read operations or to update operations. The test causes a crash of mserver5.exe after a few minutes. If the test uses just one thread, all read and update operations are done in serial and the server does not crash. What is the rule of thumb to avoid crashes when using several client threads? Crash details: Problem signature: Problem Event Name: APPCRASH Application Name: mserver5.exe Application Version: 0.0.0.0 Application Timestamp: 55389f58 Fault Module Name: StackHash_8ec6 Fault Module Version: 6.1.7601.18247 Fault Module Timestamp: 521eaf24 Exception Code: c0000374 Exception Offset: 00000000000c4102 OS Version: 6.1.7601.2.1.0.144.8 Locale ID: 1031 Additional Information 1: 8ec6 Additional Information 2: 8ec62499039d1d8234b025a62c725eb7 Additional Information 3: f8f3 Additional Information 4: f8f3e520847c0684b15d4080171779ef # MonetDB 5 server v11.19.9 "Oct2014-SP2" # Serving database 'uniserv', using 4 threads # Compiled for x86_64-pc-winnt/64bit with 64bit OIDs dynamically linked # Found 10.000 GiB available main-memory. .. # MonetDB/SQL module loaded Michael Witt _______________________________________________ users-list mailing list users-list(a)monetdb.org <mailto:users-list@monetdb.org> https://www.monetdb.org/mailman/listinfo/users-list [View Less]

2 1

Field conversion sticks to subsequent queries
by Dennis Lorson 08 Jul '15

08 Jul '15

Hi all, I have a table with an INT column. The table is sorted by that column, and monetdb has recognized the sort order. While performing this query: -------------- SELECT * FROM table WHERE column = 12345678900; -------------- monetdb converts <table.column> to BIGINT/LNG as evidenced by the trace: -------------- 3612684 | X_17=<tmp_2352>[89617785] := batcalc.lng(X_16=<tmp_7142>[89617785]); -------------- This seems logical behavior since the condition value would … [View More]

4 4

INNER JOIN with NULL values returning incorrect result
by Dennis Pallett 02 Jul '15

02 Jul '15

Hi all, I'm trying to do a simple INNER JOIN with two tables whereby the 1st table can contain NULL values in the join column. The query is as such: ---------- SELECT "B"."id", "A"."other_id" FROM "A" INNER JOIN "B" ON "B"."id" = "A"."other_id" WHERE "A"."afnemer" = '789' AND "A"."orgid" = 123; ---------- However this returns an incorrect result, such as: ---------- +------+----------+ | id | other_id | +======+==========+ | 143 | 143 | | 166 | 166 | | 147 | 147 | … [View More]

2 3

Can't start monetdb 11.19.15
by Brian Hood 01 Jul '15

01 Jul '15

Hi All, I seems when i look at the output i of the error log it seems there are missing methods, i have seen this as a bug report from a while ago online but i can't see the see the solution. Any ideas ? [root@dev-vnc-01 /]# monetdbd start /var/monetdb5/dbfarm [root@dev-vnc-01 /]# [root@dev-vnc-01 /]# monetdb status name state health remarks threatmonitor S 5d 100% 7s mapi:monetdb://dev-vnc-01:50000/threatmonitor [root@dev-vnc-01 /]# monetdb start … [View More]threatmonitor starting database 'threatmonitor'... FAILED start: starting 'threatmonitor' failed: database 'threatmonitor' started up, but failed to open up a communication channel [root@dev-vnc-01 /]# pacman -Q monetdb monetdb 11.19.15-1 [root@dev-vnc-01 /]# monetdb -v MonetDB Database Server Toolkit v1.1 (Oct2014-SP4) [root@dev-vnc-01 /]# tail -70 /var/monetdb5/dbfarm/merovingian.log 2015-06-25 23:31:58 MSG merovingian[293]: monitoring dbfarm /var/monetdb5/dbfarm 2015-06-25 23:32:18 MSG merovingian[293]: accepting connections on TCP socket 0.0.0.0:50000 2015-06-25 23:32:18 MSG merovingian[293]: accepting connections on UNIX domain socket /tmp/.s.monetdb.50000 2015-06-25 23:32:38 MSG discovery[293]: listening for UDP messages on 0.0.0.0:50000 2015-06-25 23:32:38 MSG control[293]: accepting connections on UNIX domain socket /tmp/.s.merovingian.50000 2015-06-25 23:32:38 MSG discovery[293]: new neighbour dev-vnc-01 (dev-vnc-01) 2015-06-25 23:32:53 MSG control[293]: (local): created database 'threatmonitor' 2015-06-25 23:32:59 MSG control[293]: (local): served status list 2015-06-25 23:32:59 MSG control[293]: (local): released database 'threatmonitor' 2015-06-25 23:32:59 MSG discovery[293]: new database mapi:monetdb://dev-vnc-01:50000/threatmonitor (ttl=660s) 2015-06-25 23:33:08 MSG merovingian[293]: starting database 'threatmonitor', up min/avg/max: 0s/0s/0s, crash average: 0.00 0.00 0.00 (0-0=0) 2015-06-25 23:33:08 MSG merovingian[293]: sending process 304 (database 'threatmonitor') the TERM signal 2015-06-25 23:33:08 MSG threatmonitor[304]: arguments: /usr/bin/mserver5 --dbpath=/var/monetdb5/dbfarm/threatmonitor --set merovingian_uri=mapi:monetdb://dev-vnc-01:50000/threatmonitor --set mapi_open=false --set mapi_port=0 --set mapi_usock=/var/monetdb5/dbfarm/threatmonitor/.mapi.sock --set monet_vault_key=/var/monetdb5/dbfarm/threatmonitor/.vaultkey --set gdk_nr_threads=2 --set max_clients=64 --set sql_optimizer=default_pipe --set monet_daemon=yes 2015-06-25 23:33:08 MSG threatmonitor[304]: # MonetDB 5 server v11.19.15 "Oct2014-SP4" 2015-06-25 23:33:08 MSG threatmonitor[304]: # Serving database 'threatmonitor', using 2 threads 2015-06-25 23:33:08 MSG threatmonitor[304]: # Compiled for x86_64-unknown-linux-gnu/64bit with 64bit OIDs dynamically linked 2015-06-25 23:33:08 MSG threatmonitor[304]: # Found 7.411 GiB available main-memory. 2015-06-25 23:33:08 MSG threatmonitor[304]: # Copyright (c) 1993-July 2008 CWI. 2015-06-25 23:33:08 MSG threatmonitor[304]: # Copyright (c) August 2008-2015 MonetDB B.V., all rights reserved 2015-06-25 23:33:08 MSG threatmonitor[304]: # Visit http://www.monetdb.org/ for further information 2015-06-25 23:33:08 MSG threatmonitor[304]: !TypeException:user.main[1412]:object code for command rapi.prelude missing 2015-06-25 23:33:08 MSG threatmonitor[304]: !TypeException:user.main[1412]:'rapi.prelude' undefined in: _1412:void := rapi.prelude() 2015-06-25 23:33:08 MSG merovingian[293]: database 'threatmonitor' (304) has exited with exit status 0 2015-06-25 23:33:08 MSG merovingian[293]: database 'threatmonitor' has shut down 2015-06-25 23:33:08 ERR merovingian[293]: client error: database 'threatmonitor' started up, but failed to open up a communication channel 2015-06-25 23:33:08 MSG merovingian[293]: starting database 'threatmonitor', up min/avg/max: 0s/0s/0s, crash average: 0.00 0.00 0.00 (1-1=0) 2015-06-25 23:33:09 MSG merovingian[293]: sending process 307 (database 'threatmonitor') the TERM signal 2015-06-25 23:33:09 MSG threatmonitor[307]: arguments: /usr/bin/mserver5 --dbpath=/var/monetdb5/dbfarm/threatmonitor --set merovingian_uri=mapi:monetdb://dev-vnc-01:50000/threatmonitor --set mapi_open=false --set mapi_port=0 --set mapi_usock=/var/monetdb5/dbfarm/threatmonitor/.mapi.sock --set monet_vault_key=/var/monetdb5/dbfarm/threatmonitor/.vaultkey --set gdk_nr_threads=2 --set max_clients=64 --set sql_optimizer=default_pipe --set monet_daemon=yes 2015-06-25 23:33:09 MSG threatmonitor[307]: # MonetDB 5 server v11.19.15 "Oct2014-SP4" 2015-06-25 23:33:09 MSG threatmonitor[307]: # Serving database 'threatmonitor', using 2 threads 2015-06-25 23:33:09 MSG threatmonitor[307]: # Compiled for x86_64-unknown-linux-gnu/64bit with 64bit OIDs dynamically linked 2015-06-25 23:33:09 MSG threatmonitor[307]: # Found 7.411 GiB available main-memory. 2015-06-25 23:33:09 MSG threatmonitor[307]: # Copyright (c) 1993-July 2008 CWI. 2015-06-25 23:33:09 MSG threatmonitor[307]: # Copyright (c) August 2008-2015 MonetDB B.V., all rights reserved 2015-06-25 23:33:09 MSG threatmonitor[307]: # Visit http://www.monetdb.org/ for further information 2015-06-25 23:33:09 MSG threatmonitor[307]: !TypeException:user.main[1412]:object code for command rapi.prelude missing 2015-06-25 23:33:09 MSG threatmonitor[307]: !TypeException:user.main[1412]:'rapi.prelude' undefined in: _1412:void := rapi.prelude() 2015-06-25 23:33:09 MSG merovingian[293]: database 'threatmonitor' (307) has exited with exit status 0 2015-06-25 23:33:09 MSG merovingian[293]: database 'threatmonitor' has shut down 2015-06-25 23:33:13 ERR merovingian[293]: client error: database 'threatmonitor' started up, but failed to open up a communication channel 2015-06-26 00:08:20 MSG control[293]: (local): served status list 2015-06-26 00:08:20 ERR control[293]: (local): received stop signal for non running database: threatmonitor 2015-06-26 00:08:29 MSG merovingian[293]: caught SIGTERM, starting shutdown sequence 2015-06-26 00:08:30 MSG control[293]: control channel closed 2015-06-26 00:08:31 MSG merovingian[293]: Merovingian 1.7 stopped 2015-07-01 12:57:10 MSG merovingian[9]: Merovingian 1.7 (Oct2014-SP4) starting 2015-07-01 12:57:10 MSG merovingian[9]: monitoring dbfarm /var/monetdb5/dbfarm 2015-07-01 12:57:30 MSG merovingian[9]: accepting connections on TCP socket 0.0.0.0:50000 2015-07-01 12:57:30 MSG merovingian[9]: accepting connections on UNIX domain socket /tmp/.s.monetdb.50000 2015-07-01 12:57:50 MSG discovery[9]: listening for UDP messages on 0.0.0.0:50000 2015-07-01 12:57:50 MSG control[9]: accepting connections on UNIX domain socket /tmp/.s.merovingian.50000 2015-07-01 12:57:50 MSG discovery[9]: new neighbour dev-vnc-01 (dev-vnc-01) 2015-07-01 12:57:50 MSG discovery[9]: new database mapi:monetdb://dev-vnc-01:50000/threatmonitor (ttl=660s) 2015-07-01 12:57:54 MSG control[9]: (local): served status list 2015-07-01 12:58:03 MSG control[9]: (local): served status list 2015-07-01 12:58:03 MSG merovingian[9]: starting database 'threatmonitor', up min/avg/max: 0s/7s/27s, crash average: 0.00 0.00 0.00 (10-10=0) 2015-07-01 12:58:03 MSG merovingian[9]: sending process 15 (database 'threatmonitor') the TERM signal 2015-07-01 12:58:03 MSG threatmonitor[15]: arguments: /usr/bin/mserver5 --dbpath=/var/monetdb5/dbfarm/threatmonitor --set merovingian_uri=mapi:monetdb://dev-vnc-01:50000/threatmonitor --set mapi_open=false --set mapi_port=0 --set mapi_usock=/var/monetdb5/dbfarm/threatmonitor/.mapi.sock --set monet_vault_key=/var/monetdb5/dbfarm/threatmonitor/.vaultkey --set gdk_nr_threads=2 --set max_clients=64 --set sql_optimizer=default_pipe --set monet_daemon=yes 2015-07-01 12:58:03 MSG threatmonitor[15]: # MonetDB 5 server v11.19.15 "Oct2014-SP4" 2015-07-01 12:58:03 MSG threatmonitor[15]: # Serving database 'threatmonitor', using 2 threads 2015-07-01 12:58:03 MSG threatmonitor[15]: # Compiled for x86_64-unknown-linux-gnu/64bit with 64bit OIDs dynamically linked 2015-07-01 12:58:03 MSG threatmonitor[15]: # Found 7.411 GiB available main-memory. 2015-07-01 12:58:03 MSG threatmonitor[15]: # Copyright (c) 1993-July 2008 CWI. 2015-07-01 12:58:03 MSG threatmonitor[15]: # Copyright (c) August 2008-2015 MonetDB B.V., all rights reserved 2015-07-01 12:58:03 MSG threatmonitor[15]: # Visit http://www.monetdb.org/ for further information 2015-07-01 12:58:03 MSG threatmonitor[15]: !TypeException:user.main[1412]:object code for command rapi.prelude missing 2015-07-01 12:58:03 MSG threatmonitor[15]: !TypeException:user.main[1412]:'rapi.prelude' undefined in: _1412:void := rapi.prelude() 2015-07-01 12:58:03 MSG merovingian[9]: database 'threatmonitor' (15) has exited with exit status 0 2015-07-01 12:58:03 MSG merovingian[9]: database 'threatmonitor' has shut down 2015-07-01 12:58:03 ERR control[9]: (local): failed to fork mserver: database 'threatmonitor' started up, but failed to open up a communication channel Regards, Brian Hood [View Less]

2 3

MonetDB performance disappointing
by Martin Schwitalla 30 Jun '15

30 Jun '15

Hello, I`m doing some experiments with a database, which originally runs on PostgreSQL. But some queries take very long on this system. Especially one query type, which takes about 14s to finish. The idea was to port it to MonetDB, because the database is not that big (~8GB) and it`s an analytical task. But suprisingly MonetDB takes even longer to finish. With the default pipeline it takes about 22-23s. I tried various pipelines and the minimal_pipe was the best. It finished the task in about … [View More]

5 11

Concurrency test causes crash of MonetDB 5 server
by michael.witt＠uniserv.com 30 Jun '15

30 Jun '15

Hi, we experience a crash of MonetDB 5 server during tests of a multi-threaded client access via JDBC-Driver and MapiSocket: The application is already optimized to avoid "concurrency conflicts" because of the "Optimistic Concurrency Control" concept. The test simulates client access via two threads. All update operations (INSERT, UPDATE, DELETE, DROP TABLE, CREATE TABLE) are done in serial. All read operations are done in parallel to other read operations or to update operations. The test … [View More]

2 2

JDBC PreparedStatement crashing the Database!!
by Vijay Krishna 29 Jun '15

29 Jun '15

Hi, I recently faced an issue with MonetDB JDBC PreparedStatement. I have a table with 15 columns out of which 14 are foreign keys to columns in another 14 tables. When I try to insert a valid record (that satisfies all the FK relations) using PreparedStatement, the database gets crashed while executing the line, preparedStatement = connection.prepareStatement(sqlString); On trial and error testing, I found that this works very fine until a table has 13 FK relations. If the tables has 14 … [View More]

2 2

Is minimal_pipe enforced on long MAL plans?
by Roberto Cornacchia 22 Jun '15

22 Jun '15

Hi there, Is minimal_pipe enforced under any condition? For example when the the MAL plan generated by rel2bin is longer than a given threshold? Context: I have a pretty long query which results in a MAL plan of 33K lines. Though the query is a bit long, I still would expect a 2-3K lines MAL plan. The plan indeed contains the same pieces of code re-evaluated over and over. It first, I simply blamed commonTerms not doing its job. The thing is, the MAL plans generated with minimal_pipe and … [View More]

3 3

MonetDb Udf
by Fatma Nur Özdemir 19 Jun '15

19 Jun '15

Hello, We are using MonetDB5 server v11.19.15 “Oct2014-SP4” windows version and we want to use a function that we wrote in C programming language. However, we cannot use this user defined function within MonetDB. Maybe, this problem is about windows version because we are not sure that windows version of MonetDb supports this feature. If anybody knows something about this, please inform us. Thank you in advance. Fatma Nur [http://www.bilyoner.com/fm/mail/mailalti_Bmw-applewatch.jpg]<… [View More]

1 0