- users-list - monetdb.org

single-user / locked mode
by Roberto Cornacchia 20 Feb '17

20 Feb '17

Hi, I'm confused about the single-user mode (working with latest Dec2016 from hg). *Question/suggestion 1* 'mserver5 --single-user' can also be set with 'monetdb set singleuser=yes'. However, 'monetdb get all' doesn't show 'singleuser' as one of the options, I had to guess it myself. It would be nice if monetdb could show an actual list of available options, instead a manually compiled one. *Question 2* is 'mserver5 --single-user' expected to put the database in a locked state, equivalent to 'monetdb lock' ? If the answer is yes, then it doesn't do it consistently: - it does allow to use COPY INTO.... LOCKED (from which you would think the database is locked) but: - it doesn't show the status of the database as locked - it indeed allows new client connections Could you clarify how 'single-user' and 'locked' relate? *Question 3 *(long-ish) I am pushing, via JDBC, the entire English DBpedia via COPY INTO (without 'LOCKED'), into a dbfarm on local SSD. It is about 200GB raw data and it takes ~9 hours. Data goes through a pipeline of light pre-processing (feeding thread) and then inserted into the stream of COPY INTO (output thread). What I observe is that, at the beginning, the feeding thread is mush active, and the output thread very little. After a couple of hours, the output thread is mostly active, and the feeding thread mostly idle. I told myself that MonetDB is taking more and more time to flush appends to disk. That's where I though using 'COPY INTO .. LOCKED' would help. Not only it didn't help, but it got the symptoms worse. The output thread is almost 100% active, and the feeding thread almost 100% idle. Indeed, it is already running for 12h and still going on. Was I wrong to expect that 'LOCKED' would help? Thanks, Roberto

2 2

Option to suppress status messages when running -f mode
by Anderson, David B 17 Feb '17

17 Feb '17

All, Is there a way to suppress the output of status messages like "3000 affected rows" when using mclient and "-f csv"? I have an sql script where I keep reducing a table and inserting the rows into another table. The final sql command extracts the final result. Would be nice if the status messages could be suppressed completely or sent to stderr (a --quiet option). I could also just run two scripts, or pipe the output through grep to remove the offending lines. Thanks, Dave

3 2

Best way to force table into hot set
by Anderson, David B 13 Feb '17

13 Feb '17

All, Is there a preferred method to move an entire table into the database hot-set? I guess I can have a query which computes across all of the columns in the table, but wondered if there was something better/cleaner. Thanks, Dave

1 0

prevent mitosis from partitioning (filter) function arguments
by Roberto Cornacchia 12 Feb '17

12 Feb '17

I have implemented a custom join as a filter function. By design, it cannot work correctly when the input tables are partitioned by mitosis. How can I prevent mitosis from partitioning its inputs? Roberto

2 6

Severe performance degradation
by Dennis Pallett 10 Feb '17

10 Feb '17

Hi all, We recently upgraded to the Dec2016 release but had to downgrade immediately after experiencing a severe degradation in performance. I have attached 4 graphs of our performance monitoring (of the last 3 days). It should be fairly obvious when we upgraded and when we downgraded again. We're still looking into the root cause but it appears that it has something to do with memory leaks when using aggregations on varchar colums with a high cardinality. Has anyone else experienced the same performance degradation? Is this a known (regression) bug? Best regards, Dennis Pallett

1 1

QueryTimeout Question
by Sebastián Acevedo 09 Feb '17

09 Feb '17

Hi, there is any way to chnage the default querytimeout for a user?... i need to use this to make a restriction for long querys, if i run call sys.settimeout(1) this change is only apllied to the current session and if i login again with the same user the timeout is not longer applied. +---------+-----------------+------+------------------+------+-------+ | user | login | sess | lastcommand | quer | activ | : : : iont : : ytim : e : : : : imeo : : eout : : : : : ut : : : : +=========+=================+======+==================+======+=======+ | monetdb | 2017-02-03 19:1 | 0 | 2017-02-03 19:13 | 1 | true | : : 2:37.000000 : : :28.000000 : : : | monetdb | 2017-02-03 18:0 | 0 | 2017-02-03 18:10 | 0 | false | : : 9:47.000000 : : :18.000000 : : : | monetdb | 2017-02-03 19:1 | 0 | 2017-02-03 19:13 | 0 | false | : : 3:10.000000 : : :12.000000 : : : +---------+-----------------+------+------------------+------+-------+ Thanks Sebastián Acevedo

3 4

Python Aggregate functions
by Anderson, David B 06 Feb '17

06 Feb '17

Mark, I think there is a bug in the implementation of the PYTHON_MAP aggregate functions. The return value of a PYTHON_MAP aggregate function can change if there are other functions in the query. The return value of my function weighted_percentile_0 (definition below) changes if I include the function median() in my query. The results are incorrect if median() isn't in the query, but are correct if median() is in the query. In this test case, the weights (v2) are all 1 and the values (v1) are uniformly sampled integers from [1,10]. Number of rows for each fctr is ~1000, and my function should reduce to min(v1) in this case. Is some information bleeding between threads? Should I submit a bug report? This is on the default branch. Thanks, Dave sql>select fctr,weighted_percentile_0(v1,v2),min(v1) from mini where fctr < 10 group by fctr order by fctr; +------+--------------------------+------+ | fctr | L1 | L2 | +======+==========================+======+ | 1 | 3 | 1 | | 2 | 2 | 1 | | 3 | 2 | 1 | | 4 | 1 | 1 | | 5 | 3 | 1 | | 6 | 3 | 1 | | 7 | 2 | 1 | | 8 | 1 | 1 | | 9 | 2 | 1 | +------+--------------------------+------+ 9 tuples (126.928ms) sql>select fctr,weighted_percentile_0(v1,v2),min(v1),median(v1) from mini where fctr < 10 group by fctr order by fctr; +------+--------------------------+------+------+ | fctr | L1 | L2 | L3 | +======+==========================+======+======+ | 1 | 1 | 1 | 5 | | 2 | 1 | 1 | 5 | | 3 | 1 | 1 | 5 | | 4 | 1 | 1 | 5 | | 5 | 1 | 1 | 5 | | 6 | 1 | 1 | 5 | | 7 | 1 | 1 | 5 | | 8 | 1 | 1 | 5 | | 9 | 1 | 1 | 5 | +------+--------------------------+------+------+ 9 tuples (519.195ms) sql> CREATE AGGREGATE weighted_percentile_0(a DOUBLE, w DOUBLE) RETURNS DOUBLE LANGUAGE PYTHON_MAP { import numpy as np # Standardize and sort based on values in a q = np.array([0]) / 100.0 idx = np.argsort(a) a_sort = a[idx] w_sort = w[idx] # Get the cumulative sum of weights ecdf = np.cumsum(w_sort) # Find the percentile index positions associated with the percentiles p = q * (w_sort.sum() - 1) # Find the bounding indices (both low and high) idx_low = np.searchsorted(ecdf, p, side='right') idx_high = np.searchsorted(ecdf, p + 1, side='right') idx_high[idx_high > ecdf.size - 1] = ecdf.size - 1 # Calculate the weights weights_high = p - np.floor(p) weights_low = 1.0 - weights_high # Extract the low/high indexes and multiply by the corresponding weights x1 = np.take(a_sort, idx_low) * weights_low x2 = np.take(a_sort, idx_high) * weights_high wp = np.add(x1,x2) return(wp[0]) };

3 4

Virtual Column
by imad hajj chahine 01 Feb '17

01 Feb '17

Hi, How far is the feature of supporting virtual column, with the possibility to persist it as a physical column, from the future release of monetdb? This feature is supported in mysql starting the 5.7 version. https://dev.mysql.com/doc/refman/5.7/en/create-table-generated-columns.html Thank you.

1 0

SQL grammar
by imad hajj chahine 01 Feb '17

01 Feb '17

Hi, I am interested to know how actually monetdb parse an SQL expression and generate the expression tree, are you using lex/yacc similar grammar tool. I am currently using antlr for some basic parsing tasks, but my objective is to be able to generate a complete expression tree from an SQL expression. Thank you.

1 0

Installing Monetdb in on Ubuntu 16.04.1 LTS via repo
by Yan Grange 01 Feb '17

01 Feb '17

Hi there, I'm trying to install monetdb via the instructions on the website inside a Docker container derived from the Ubuntu 16.04.1 LTS image. I get the following error message: root@ContainerHost:/local# sudo apt-get update Ign:1 http://dev.monetdb.org/downloads/deb xenial InRelease Get:2 http://dev.monetdb.org/downloads/deb xenial Release [3878 B] Get:3 http://dev.monetdb.org/downloads/deb xenial Release.gpg [163 B] Hit:4 http://archive.ubuntu.com/ubuntu xenial InRelease Hit:5 http://archive.ubuntu.com/ubuntu xenial-updates InRelease Hit:6 http://archive.ubuntu.com/ubuntu xenial-security InRelease Fetched 4041 B in 0s (6161 B/s) Reading package lists... Done W: http://dev.monetdb.org/downloads/deb/dists/xenial/Release.gpg: Signature by key 213E64DCD5DDC2C063CC39FD053C3ED40583366F uses weak digest algorithm (SHA1) W: Invalid 'Date' entry in Release file /var/lib/apt/lists/partial/dev.monetdb.org_downloads_deb_dists_xenial_Release E: Failed to fetch http://dev.monetdb.org/downloads/deb/dists/xenial/Release Invalid 'Valid-Until' entry in Release file /var/lib/apt/lists/partial/dev.monetdb.org_downloads_deb_dists_xenial_Release E: Some index files failed to download. They have been ignored, or old ones used instead. I did only find one reference to this in a previous discussion. The advice given was to create a bug report. Since for that I need to have an account and that I can't find an account creation link on the bugzilla page, I decided to use this mailing list. Regards and thanks a lot, Yan Grange

2 1

problem stopping/starting monetdbd
by Lynn Carol Johnson 30 Jan '17

30 Jan '17

Hi all - Yesterday our monetdb instance seemed to be running slowing, perhaps due to user queries that had been killed but maybe not cleaned up. We decided to stop the dbs, then start/stop monetdbd. I executed the following commands, but was unable to restart the daemon: sudo monetdbd stop /local/monetdb/dbfarmBL sudo monetdbd start /local/monetdb/dbfarmBL We had our admin reboot the machine, thinking this would clear up everything. This morning I tried again to start monetdbd. The command comes back with no messages, but I get a connection refused message when trying to access a database: could not connect to localhost:50000: Connection refused Get all shows this: [lcj34@cbsudc01 anp68_ije_Locus9Combined_Result]$ monetdbd get all /local/monetdb/dbfarmBL property value hostname cbsudc01 dbfarm /local/monetdb/dbfarmBL status no monetdbd is serving this dbfarm mserver unknown (monetdbd not running) logfile /local/monetdb/dbfarmBL/merovingian.log pidfile /local/monetdb/dbfarmBL/merovingian.pid sockdir /tmp listenaddr 128.84.3.206 port 50000 exittimeout 60 forward proxy discovery yes discoveryttl 600 control no passphrase <unknown> mapisock /tmp/.s.monetdb.50000 controlsock /tmp/.s.merovingian.50000 [lcj34@cbsudc01 anp68_ije_Locus9Combined_Result]$ The log shows the following message: 2017-01-27 07:53:26 MSG maizeFullGenomeDB[14286]: # Listening for UNIX domain connection requests on mapi:monetdb:///local/monetdb/dbfarmBL/maizeFullGenomeDB/.mapi.sock 2017-01-27 07:53:26 MSG maizeFullGenomeDB[14286]: # MonetDB/SQL module loaded 2017-01-27 07:53:29 ERR merovingian[13705]: client error: cannot connect: Connection refused 2017-01-27 07:55:41 MSG merovingian[13705]: caught SIGTERM, starting shutdown sequence 2017-01-27 07:55:50 MSG merovingian[15395]: Merovingian 1.7 (Jun2016-SP2) starting 2017-01-27 07:55:50 MSG merovingian[15395]: monitoring dbfarm /local/monetdb/dbfarmBL 2017-01-27 07:55:50 MSG merovingian[15395]: accepting connections on TCP socket cbsudc01:50000 2017-01-27 07:55:50 MSG merovingian[15395]: accepting connections on UNIX domain socket /tmp/.s.monetdb.50000 2017-01-27 07:55:50 ERR merovingian[15395]: binding to datagram socket port 50000 failed: no available address 2017-01-27 07:55:50 MSG merovingian[15395]: Merovingian 1.7 stopped 2017-01-27 07:55:50 ERR merovingian[15395]: fatal startup condition encountered, aborting startup Any ideas what to try? Thanks - Lynn

3 7

Re: running make command gives error on installation
by Pham Duc 29 Jan '17

29 Jan '17

Hi Miguel, I guess that you tried MonetDB 11.2.0 as it is mentioned in the "Heuristics-based Query Optimisation for SPARQL" paper (https://openproceedings.org/2012/conf/edbt/TsialiamanisSFCB12.pdf) Did you check the instruction provided by the ICS-FORTH researchers ( http://www.ics.forth.gr/isl/index_main.php?l=e&c=646 ) for installing MonetDB? If not, please follow that instruction. If it still does not work, you will need to contact ICS-FORTH researchers (please see the contact at the end of the instruction) as that RDF version of MonetDB 11.2.0 was specifically developed for their research with the support of our group members (e.g., Lefteris). Best regards, Minh Duc ----- Original Message ----- From: "Martin Kersten" <martin.kersten(a)cwi.nl> To: "Communication channel for MonetDB users" <users-list(a)monetdb.org>, "P Boncz" <P.Boncz(a)cwi.nl>, "Pham Minh Duc" <P.Minh.Duc(a)cwi.nl> Sent: Sunday, January 29, 2017 8:41:12 AM Subject: Re: running make command gives error on installation Hi Indeed, this is a branch used by Pham Duc during his PhD research. The code base has not been kept in sync with the main branches and you should contact him/his advisor for further support. regards, Martin On 28/01/2017 21:59, Miguel Mármol wrote: > Hi, > I am trying to install MonetDB5 11.2.0 (The version of MonetDB is not an official MonetDB release. This version includes an RDF frontend that is used to parse RDF datasets and load them in MonetDB.). > > I get the following error when I run the make command: > > > miguel@miguel-VirtualBox:~/Downloads/MonetDB$ sudo make > make all-recursive > make[1]: Entering directory '/home/miguel/Downloads/MonetDB' > Making all in buildtools > make[2]: Entering directory '/home/miguel/Downloads/MonetDB/buildtools' > make all-recursive > make[3]: Entering directory '/home/miguel/Downloads/MonetDB/buildtools' > Making all in Mx > make[4]: Entering directory '/home/miguel/Downloads/MonetDB/buildtools/Mx' > make all-am > make[5]: Entering directory '/home/miguel/Downloads/MonetDB/buildtools/Mx' > make[5]: Nothing to be done for 'all-am'. > make[5]: Leaving directory '/home/miguel/Downloads/MonetDB/buildtools/Mx' > make[4]: Leaving directory '/home/miguel/Downloads/MonetDB/buildtools/Mx' > Making all in conf > make[4]: Entering directory '/home/miguel/Downloads/MonetDB/buildtools/conf' > make all-am > make[5]: Entering directory '/home/miguel/Downloads/MonetDB/buildtools/conf' > make[5]: Nothing to be done for 'all-am'. > make[5]: Leaving directory '/home/miguel/Downloads/MonetDB/buildtools/conf' > make[4]: Leaving directory '/home/miguel/Downloads/MonetDB/buildtools/conf' > make[4]: Entering directory '/home/miguel/Downloads/MonetDB/buildtools' > make[4]: Nothing to be done for 'all-am'. > make[4]: Leaving directory '/home/miguel/Downloads/MonetDB/buildtools' > make[3]: Leaving directory '/home/miguel/Downloads/MonetDB/buildtools' > make[2]: Leaving directory '/home/miguel/Downloads/MonetDB/buildtools' > Making all in common > make[2]: Entering directory '/home/miguel/Downloads/MonetDB/common' > make all-recursive > make[3]: Entering directory '/home/miguel/Downloads/MonetDB/common' > Making all in stream > make[4]: Entering directory '/home/miguel/Downloads/MonetDB/common/stream' > make all-am > make[5]: Entering directory '/home/miguel/Downloads/MonetDB/common/stream' > make[5]: Nothing to be done for 'all-am'. > make[5]: Leaving directory '/home/miguel/Downloads/MonetDB/common/stream' > make[4]: Leaving directory '/home/miguel/Downloads/MonetDB/common/stream' > Making all in options > make[4]: Entering directory '/home/miguel/Downloads/MonetDB/common/options' > make all-am > make[5]: Entering directory '/home/miguel/Downloads/MonetDB/common/options' > make[5]: Nothing to be done for 'all-am'. > make[5]: Leaving directory '/home/miguel/Downloads/MonetDB/common/options' > make[4]: Leaving directory '/home/miguel/Downloads/MonetDB/common/options' > Making all in utils > make[4]: Entering directory '/home/miguel/Downloads/MonetDB/common/utils' > make all-am > make[5]: Entering directory '/home/miguel/Downloads/MonetDB/common/utils' > make[5]: Nothing to be done for 'all-am'. > make[5]: Leaving directory '/home/miguel/Downloads/MonetDB/common/utils' > make[4]: Leaving directory '/home/miguel/Downloads/MonetDB/common/utils' > make[4]: Entering directory '/home/miguel/Downloads/MonetDB/common' > make[4]: Nothing to be done for 'all-am'. > make[4]: Leaving directory '/home/miguel/Downloads/MonetDB/common' > make[3]: Leaving directory '/home/miguel/Downloads/MonetDB/common' > make[2]: Leaving directory '/home/miguel/Downloads/MonetDB/common' > Making all in clients > make[2]: Entering directory '/home/miguel/Downloads/MonetDB/clients' > make all-recursive > make[3]: Entering directory '/home/miguel/Downloads/MonetDB/clients' > Making all in mapilib > make[4]: Entering directory '/home/miguel/Downloads/MonetDB/clients/mapilib' > make all-am > make[5]: Entering directory '/home/miguel/Downloads/MonetDB/clients/mapilib' > make[5]: Nothing to be done for 'all-am'. > make[5]: Leaving directory '/home/miguel/Downloads/MonetDB/clients/mapilib' > make[4]: Leaving directory '/home/miguel/Downloads/MonetDB/clients/mapilib' > Making all in mapiclient > make[4]: Entering directory '/home/miguel/Downloads/MonetDB/clients/mapiclient' > make all-am > make[5]: Entering directory '/home/miguel/Downloads/MonetDB/clients/mapiclient' > /bin/bash ../../libtool --tag=CC --mode=link gcc -Wall -Wextra -g -pedantic -Wno-long-long -Wno-declaration-after-statement -D_REENTRANT -export-dynamic -o mclient mclient.o ReadlineTools.o libmcutil.la ../mapilib/libmapi.la > ../../common/stream/libstream.la -lcrypt -pthread > libtool: link: gcc -Wall -Wextra -g -pedantic -Wno-long-long -Wno-declaration-after-statement -D_REENTRANT -o .libs/mclient mclient.o ReadlineTools.o -pthread -Wl,--export-dynamic ./.libs/libmcutil.a ../mapilib/.libs/libmapi.so > ../../common/stream/.libs/libstream.so -lcrypt -pthread -Wl,-rpath -Wl,/usr/local/monet/lib > ../mapilib/.libs/libmapi.so: undefined reference to `SHA256' > ../mapilib/.libs/libmapi.so: undefined reference to `SHA384' > ../mapilib/.libs/libmapi.so: undefined reference to `RIPEMD160' > ../mapilib/.libs/libmapi.so: undefined reference to `MD5' > ../mapilib/.libs/libmapi.so: undefined reference to `SHA1' > ../mapilib/.libs/libmapi.so: undefined reference to `SHA512' > ../mapilib/.libs/libmapi.so: undefined reference to `SHA224' > collect2: error: ld returned 1 exit status > Makefile:666: recipe for target 'mclient' failed > make[5]: *** [mclient] Error 1 > make[5]: Leaving directory '/home/miguel/Downloads/MonetDB/clients/mapiclient' > Makefile:568: recipe for target 'all' failed > make[4]: *** [all] Error 2 > make[4]: Leaving directory '/home/miguel/Downloads/MonetDB/clients/mapiclient' > Makefile:764: recipe for target 'all-recursive' failed > make[3]: *** [all-recursive] Error 1 > make[3]: Leaving directory '/home/miguel/Downloads/MonetDB/clients' > Makefile:709: recipe for target 'all' failed > make[2]: *** [all] Error 2 > make[2]: Leaving directory '/home/miguel/Downloads/MonetDB/clients' > Makefile:600: recipe for target 'all-recursive' failed > make[1]: *** [all-recursive] Error 1 > make[1]: Leaving directory '/home/miguel/Downloads/MonetDB' > Makefile:530: recipe for target 'all' failed > make: *** [all] Error 2 > > Please any help will be really appreciated, thanks. > > Miguel > > > > _______________________________________________ > users-list mailing list > users-list(a)monetdb.org > https://www.monetdb.org/mailman/listinfo/users-list >

1 0

running make command gives error on installation
by Miguel Mármol 29 Jan '17

29 Jan '17

Hi, I am trying to install MonetDB5 11.2.0 (The version of MonetDB is not an official MonetDB release. This version includes an RDF frontend that is used to parse RDF datasets and load them in MonetDB.). I get the following error when I run the make command: miguel@miguel-VirtualBox:~/Downloads/MonetDB$ sudo make make all-recursive make[1]: Entering directory '/home/miguel/Downloads/MonetDB' Making all in buildtools make[2]: Entering directory '/home/miguel/Downloads/MonetDB/buildtools' make all-recursive make[3]: Entering directory '/home/miguel/Downloads/MonetDB/buildtools' Making all in Mx make[4]: Entering directory '/home/miguel/Downloads/MonetDB/buildtools/Mx' make all-am make[5]: Entering directory '/home/miguel/Downloads/MonetDB/buildtools/Mx' make[5]: Nothing to be done for 'all-am'. make[5]: Leaving directory '/home/miguel/Downloads/MonetDB/buildtools/Mx' make[4]: Leaving directory '/home/miguel/Downloads/MonetDB/buildtools/Mx' Making all in conf make[4]: Entering directory '/home/miguel/Downloads/MonetDB/buildtools/conf' make all-am make[5]: Entering directory '/home/miguel/Downloads/MonetDB/buildtools/conf' make[5]: Nothing to be done for 'all-am'. make[5]: Leaving directory '/home/miguel/Downloads/MonetDB/buildtools/conf' make[4]: Leaving directory '/home/miguel/Downloads/MonetDB/buildtools/conf' make[4]: Entering directory '/home/miguel/Downloads/MonetDB/buildtools' make[4]: Nothing to be done for 'all-am'. make[4]: Leaving directory '/home/miguel/Downloads/MonetDB/buildtools' make[3]: Leaving directory '/home/miguel/Downloads/MonetDB/buildtools' make[2]: Leaving directory '/home/miguel/Downloads/MonetDB/buildtools' Making all in common make[2]: Entering directory '/home/miguel/Downloads/MonetDB/common' make all-recursive make[3]: Entering directory '/home/miguel/Downloads/MonetDB/common' Making all in stream make[4]: Entering directory '/home/miguel/Downloads/MonetDB/common/stream' make all-am make[5]: Entering directory '/home/miguel/Downloads/MonetDB/common/stream' make[5]: Nothing to be done for 'all-am'. make[5]: Leaving directory '/home/miguel/Downloads/MonetDB/common/stream' make[4]: Leaving directory '/home/miguel/Downloads/MonetDB/common/stream' Making all in options make[4]: Entering directory '/home/miguel/Downloads/MonetDB/common/options' make all-am make[5]: Entering directory '/home/miguel/Downloads/MonetDB/common/options' make[5]: Nothing to be done for 'all-am'. make[5]: Leaving directory '/home/miguel/Downloads/MonetDB/common/options' make[4]: Leaving directory '/home/miguel/Downloads/MonetDB/common/options' Making all in utils make[4]: Entering directory '/home/miguel/Downloads/MonetDB/common/utils' make all-am make[5]: Entering directory '/home/miguel/Downloads/MonetDB/common/utils' make[5]: Nothing to be done for 'all-am'. make[5]: Leaving directory '/home/miguel/Downloads/MonetDB/common/utils' make[4]: Leaving directory '/home/miguel/Downloads/MonetDB/common/utils' make[4]: Entering directory '/home/miguel/Downloads/MonetDB/common' make[4]: Nothing to be done for 'all-am'. make[4]: Leaving directory '/home/miguel/Downloads/MonetDB/common' make[3]: Leaving directory '/home/miguel/Downloads/MonetDB/common' make[2]: Leaving directory '/home/miguel/Downloads/MonetDB/common' Making all in clients make[2]: Entering directory '/home/miguel/Downloads/MonetDB/clients' make all-recursive make[3]: Entering directory '/home/miguel/Downloads/MonetDB/clients' Making all in mapilib make[4]: Entering directory '/home/miguel/Downloads/MonetDB/clients/mapilib' make all-am make[5]: Entering directory '/home/miguel/Downloads/MonetDB/clients/mapilib' make[5]: Nothing to be done for 'all-am'. make[5]: Leaving directory '/home/miguel/Downloads/MonetDB/clients/mapilib' make[4]: Leaving directory '/home/miguel/Downloads/MonetDB/clients/mapilib' Making all in mapiclient make[4]: Entering directory '/home/miguel/Downloads/MonetDB/clients/mapiclient' make all-am make[5]: Entering directory '/home/miguel/Downloads/MonetDB/clients/mapiclient' /bin/bash ../../libtool --tag=CC --mode=link gcc -Wall -Wextra -g -pedantic -Wno-long-long -Wno-declaration-after-statement -D_REENTRANT -export-dynamic -o mclient mclient.o ReadlineTools.o libmcutil.la ../mapilib/libmapi.la ../../common/stream/libstream.la -lcrypt -pthread libtool: link: gcc -Wall -Wextra -g -pedantic -Wno-long-long -Wno-declaration-after-statement -D_REENTRANT -o .libs/mclient mclient.o ReadlineTools.o -pthread -Wl,--export-dynamic ./.libs/libmcutil.a ../mapilib/.libs/libmapi.so ../../common/stream/.libs/libstream.so -lcrypt -pthread -Wl,-rpath -Wl,/usr/local/monet/lib ../mapilib/.libs/libmapi.so: undefined reference to `SHA256' ../mapilib/.libs/libmapi.so: undefined reference to `SHA384' ../mapilib/.libs/libmapi.so: undefined reference to `RIPEMD160' ../mapilib/.libs/libmapi.so: undefined reference to `MD5' ../mapilib/.libs/libmapi.so: undefined reference to `SHA1' ../mapilib/.libs/libmapi.so: undefined reference to `SHA512' ../mapilib/.libs/libmapi.so: undefined reference to `SHA224' collect2: error: ld returned 1 exit status Makefile:666: recipe for target 'mclient' failed make[5]: *** [mclient] Error 1 make[5]: Leaving directory '/home/miguel/Downloads/MonetDB/clients/mapiclient' Makefile:568: recipe for target 'all' failed make[4]: *** [all] Error 2 make[4]: Leaving directory '/home/miguel/Downloads/MonetDB/clients/mapiclient' Makefile:764: recipe for target 'all-recursive' failed make[3]: *** [all-recursive] Error 1 make[3]: Leaving directory '/home/miguel/Downloads/MonetDB/clients' Makefile:709: recipe for target 'all' failed make[2]: *** [all] Error 2 make[2]: Leaving directory '/home/miguel/Downloads/MonetDB/clients' Makefile:600: recipe for target 'all-recursive' failed make[1]: *** [all-recursive] Error 1 make[1]: Leaving directory '/home/miguel/Downloads/MonetDB' Makefile:530: recipe for target 'all' failed make: *** [all] Error 2 Please any help will be really appreciated, thanks. Miguel

2 1

apt having trouble installing deb packages
by Clayton A Davis 27 Jan '17

27 Jan '17

Ubuntu seems to be having an issue with the deb packages. On 16.04, it complains about the "Valid-Until" entry in the Release file, and on 16.10 it complains about the "Date" entry and the security. Here are the error messages from 16.04: W: http://dev.monetdb.org/downloads/deb/dists/xenial/Release.gpg: Signature by key 213E64DCD5DDC2C063CC39FD053C3ED40583366F uses weak digest algorithm (SHA1) W: Invalid 'Date' entry in Release file /var/lib/apt/lists/dev.monetdb.org_downloads_deb_dists_xenial_Release W: http://www.openprinting.org/download/printdriver/debian/dists/lsb3.2/Releas…: Signature by key F8897B6F00075648E248B7EC24CBF5474CFD1E2F uses weak digest algorithm (SHA1) E: Failed to fetch http://dev.monetdb.org/downloads/deb/dists/xenial/Release Invalid 'Valid-Until' entry in Release file /var/lib/apt/lists/dev.monetdb.org_downloads_deb_dists_xenial_Release And from 16.10: W: http://dev.monetdb.org/downloads/deb/dists/yakkety/Release.gpg: Signature by key 213E64DCD5DDC2C063CC39FD053C3ED40583366F uses weak digest algorithm (SHA1) W: Invalid 'Date' entry in Release file /var/lib/apt/lists/partial/dev.monetdb.org_downloads_deb_dists_yakkety_Release E: The repository 'http://dev.monetdb.org/downloads/deb yakkety Release' provides only weak security information. N: Updating from such a repository can't be done securely, and is therefore disabled by default. N: See apt-secure(8) manpage for repository creation and user configuration details. I know the workarounds to get it working on my machine, I just wanted to put this issue out there in case it started with the recent release. -CAD

1 0

Announcement: New Dec2016-SP1 Bugfix release of MonetDB suite
by Panagiotis Koutsourakis 26 Jan '17

26 Jan '17

The MonetDB team at CWI/MonetDB BV is pleased to announce the Dec2016-SP1 bugfix release of the MonetDB suite of programs. More information about MonetDB can be found on our website at <http://www.monetdb.org/>. For details on this release, please see the release notes at <http://www.monetdb.org/Downloads/ReleaseNotes>. As usual, the download location is <http://dev.monetdb.org/downloads/>. Dec 2016-SP1 bugfix release Bug Fixes * 4039: Slow mserver5 start after drop of tables (> 1 hour) * 4048: Segfault on vacuum with parallel updates * 6079: pushselect optimizer bug on MAL snippet * 6140: INNER JOIN gives the results of a CROSS JOIN * 6150: Query giving wrong results, extra records are appearing * 6175: The program can't start because python27.dll is missing from your computer. * 6178: AVG + GROUP BY returns NULL for some records that should have results * 6179: mergetable optimizer messes up sample * 6182: sys.shutdown triggers assertion in clients.c * 6184: Incorrect result set - Extra records in result set

1 0

Backup Strategies
by Vincent Sheffer 25 Jan '17

25 Jan '17

I have been attempting to create a viable period backup strategy for MonetDB and have had little success to date. Our database is way too big to use msqldump, so our only option is the one outlined here: https://www.monetdb.org/Documentation/UserGuide/FastDumpRestore. I follow those steps and use rsync to do an incremental backup of the dbfarm to another device (from fast SSD to RAID5 array). That works fine, until it doesn’t, in which case the back up gets corrupted somehow and MonetDB won’t start from the backup. That is the state I am currently in. This is the error that I am getting: !FATAL: logger_load: BBPrename to sql_snapshots_bid failed Any help on 1. Fixing the existing backup, and/or 2. Helping me understand a better way to do period, incremental backups. Thanks, Vince The information transmitted, including any attachments, is intended only for the individual or entity to which it is addressed, and may contain confidential and/or privileged information. Any review, retransmission, dissemination or other use of, or taking of any action in reliance upon, this information by individuals or entities other than the intended recipient is prohibited, and all liability arising therefrom is disclaimed. If you have received this communication in error, please delete the information from any computer and notify the sender.

4 8

Database crash on executing a big insert
by Shrihari Shastry 25 Jan '17

25 Jan '17

Hi, I' trying to perform a big insert of about 4 mil rows of data: insert into clustertable1(keyword, p0,p1,p2,p3,p4,p5,p6,p7,p8,p9,p10,p11,p12,p13,p14,p15,p16,p17,p18,p19,p20,p21,p22,p23,p24,p25,p26,p27,p28,p29,p30,p31,p32,p33,p34,p35,p36,p37,p38,p39,p40,p41,p42,p43,p44,p45,p46,p47,p48,p49,p50) select keyword,p0,p1,p2,p3,p4,p5,p6,p7,p8,p9,p10,p11,p12,p13,p14,p15,p16,p17,p18,p19,p20,p21,p22,p23,p24,p25,p26,p27,p28,p29,p30,p31,p32,p33,p34,p35,p36,p37,p38,p39,p40,p41,p42,p43,p44,p45,p46,p47,p48,p49,p50 from cluster_submaster where cluster_no = '0'; But the moment I fire this query, the server crashes with the below printed in the logs: 2017-01-25 10:05:11 MSG kwresearch[3418]: # Visit http://www.monetdb.org/ for further information 2017-01-25 10:05:11 MSG kwresearch[3418]: # Listening for UNIX domain connection requests on mapi:monetdb:///monetDB/myfarm/kwresearch/.mapi.sock 2017-01-25 10:05:11 MSG kwresearch[3418]: # MonetDB/SQL module loaded 2017-01-25 10:07:06 MSG merovingian[3201]: proxying client (local) for database 'kwresearch' to mapi:monetdb:///monetDB/myfarm/kwresearch/.mapi.sock?database=kwresearch 2017-01-25 10:07:06 MSG merovingian[3201]: target connection is on local UNIX domain socket, passing on filedescriptor instead of proxying 2017-01-25 10:07:06 MSG merovingian[3201]: proxying client (local) for database 'kwresearch' to mapi:monetdb:///monetDB/myfarm/kwresearch/.mapi.sock?database=kwresearch 2017-01-25 10:07:06 MSG merovingian[3201]: target connection is on local UNIX domain socket, passing on filedescriptor instead of proxying 2017-01-25 10:07:12 MSG merovingian[3201]: database 'kwresearch' (3418) was killed by signal SIGSEGV Appreciate any help! Thanks, Shrihari ******************* iquanti.com E-Mail Domain LEGAL DISCLAIMER ******************* This E-Mail may contain confidential and/or legally privileged Information and is meant for the intended recipient(s) only. If you have received this e-mail in error and are not the intended recipient/s, kindly notify us at it(a)iquanti.com and then delete this e-mail immediately from your system. You are also hereby notified that any use, any form of reproduction, dissemination, copying, disclosure, modification, distribution and/or publication of this e-mail, its contents or its attachment/s other than by its intended recipient/s is strictly prohibited and may be unlawful.

2 1

bool from aggregate
by Stefano Fioravanzo 25 Jan '17

25 Jan '17

Hello, I would like to know if there is a way to replicate the behavior of the bool_and function of Postgresql. An example of its use is the following: SELECT bool_and(A) FROM mytable GROUP BY B; bool_and will return true if all values in the group are true, false otherwise. I would like to do the same thing in monetdb. Thanks Stefano

2 1

Case insensitive version of the IN operator?
by Kevork Vartanian 13 Jan '17

13 Jan '17

Hi, I was wondering if there is a case insensitive version of the IN operator, similar to ILIKE. Or is the IN itself case insensitive? It's mainly to avoid using nested OR's and ILIKE's if a case insensitive IN exists, to make the queries much simpler. Also, I don't want to use functions like lower as it's a big data set and the use of functions may make the queries even slower. Please advise. Thank you. Regards, Kevork Vartanian Technical Solutions Consultant kevork.vartanian(a)innotta.com.au | +61 406 776 789 | www.innotta.com.au

3 2

mclient COPY INFO Local files to a remote MonetDB Server Error
by ping gao 高平(0) 11 Jan '17

11 Jan '17

Hi ALL mclient COPY INFO Local files to a remote MonetDB Server Error. Mclient how to copy a local file to a remote MonetDB Server? Known to use: mclient COPY INFO - < file ; Will appear: syntax error, unexpected sqlINT in: Error Error Info :could not open file ‘xxxxx’ : No such file or directory

3 4

COPY INFO ERROR
by ping gao 高平(0) 10 Jan '17

10 Jan '17

Hi ALL I encountered a problem using a COPY INFO. Strange thing is my IP is 59, execute the following statement is not a problem on the machine.So freak! IP:192.168.110.59 (Database Server) [hotel-revenue-online@bjs-hotel-RevenueCompute-02 data_visual_monetdb]$ mclient -u monetdb -d hotel-revenue -s "COPY 2 RECORDS INTO hotel_revenue_octopus_booking_order FROM '/home/hotel-revenue-online/ping.gao/data_visual_monetdb/hotel_revenue_octopus_booking_order.sql' USING DELIMITERS ',','\\n'<smb://n'>" password: 2 affected rows IP:192.168.110.179 Error 1: [hotel-revenue-online@bjs-hotel-RevenueCompute-01 datavisual_service]$ mclient -h 192.168.110.59 -p 50001 -d hotel-revenue -s "COPY 2 RECORDS INTO hotel_revenue_octopus_hotel_info FROM /home/hotel-revenue-online/datavisual_service/data_tmp/hotel_revenue_octopus_hotel_info.data USING DELIMITERS '\\001','\\n'<smb://001','//n'>” syntax error, unexpected '/', expecting STRING or '(' or STDIN in: "copy 2 records into hotel_revenue_octopus_hotel_info from /“ Error 2: [hotel-revenue-online@bjs-hotel-RevenueCompute-01 datavisual_service]$ mclient -h 192.168.110.59 -p 50001 -d hotel-revenue -s "COPY 2 RECORDS INTO hotel_revenue_octopus_hotel_info FROM '/home/hotel-revenue-online/datavisual_service/data_tmp/hotel_revenue_octopus_hotel_info.data' USING DELIMITERS '\\001<smb://001>','\\n'<smb://n'>" could not open file '/home/hotel-revenue-online/datavisual_service/data_tmp/hotel_revenue_octopus_hotel_info.data': No such file or directory Error 3: [hotel-revenue-online@bjs-hotel-RevenueCompute-01 datavisual_service]$ mclient -h 192.168.110.59 -p 50001 -d hotel-revenue -s "COPY 2 RECORDS INTO hotel_revenue_octopus_hotel_info FROM STDIN DELIMITERS '\\001','\\n'<smb://001','//n'>" - < /home/hotel-revenue-online/datavisual_service/data_tmp/hotel_revenue_octopus_hotel_info.data 2 affected rows syntax error, unexpected sqlINT in: "90564318"

2 2

RE: COPY INFO ERROR
by Stefan Manegold 10 Jan '17

10 Jan '17

Not strange at all. Quotes are missing around file path (string). Sent from my Samsung Galaxy smartphone. -------- Original message --------From: "ping gao 高平(0)" <ping.gao(a)corp.elong.com> Date: 1/10/17 08:07 (GMT+01:00) To: Communication channel for MonetDB users <users-list(a)monetdb.org> Subject: COPY INFO ERROR Hi ALL I encountered a problem using a COPY INFO. Strange thing is my IP is 59, execute the following statement is not a problem on the machine.So freak! IP:192.168.110.59 (Database Server) [hotel-revenue-online@bjs-hotel-RevenueCompute-02 data_visual_monetdb]$ mclient -u monetdb -d hotel-revenue -s "COPY 2 RECORDS INTO hotel_revenue_octopus_booking_order FROM '/home/hotel-revenue-online/ping.gao/data_visual_monetdb/hotel_revenue_octopus_booking_order.sql' USING DELIMITERS ',','\\n'" password: 2 affected rows IP:192.168.110.179 Error 1: [hotel-revenue-online@bjs-hotel-RevenueCompute-01 datavisual_service]$ mclient -h 192.168.110.59 -p 50001 -d hotel-revenue -s "COPY 2 RECORDS INTO hotel_revenue_octopus_hotel_info FROM /home/hotel-revenue-online/datavisual_service/data_tmp/hotel_revenue_octopus_hotel_info.data USING DELIMITERS '\\001','\\n'” syntax error, unexpected '/', expecting STRING or '(' or STDIN in: "copy 2 records into hotel_revenue_octopus_hotel_info from /“ Error 2: [hotel-revenue-online@bjs-hotel-RevenueCompute-01 datavisual_service]$ mclient -h 192.168.110.59 -p 50001 -d hotel-revenue -s "COPY 2 RECORDS INTO hotel_revenue_octopus_hotel_info FROM '/home/hotel-revenue-online/datavisual_service/data_tmp/hotel_revenue_octopus_hotel_info.data' USING DELIMITERS '\\001','\\n'" could not open file '/home/hotel-revenue-online/datavisual_service/data_tmp/hotel_revenue_octopus_hotel_info.data': No such file or directory Error 3: [hotel-revenue-online@bjs-hotel-RevenueCompute-01 datavisual_service]$ mclient -h 192.168.110.59 -p 50001 -d hotel-revenue -s "COPY 2 RECORDS INTO hotel_revenue_octopus_hotel_info FROM STDIN DELIMITERS '\\001','\\n'" - < /home/hotel-revenue-online/datavisual_service/data_tmp/hotel_revenue_octopus_hotel_info.data 2 affected rows syntax error, unexpected sqlINT in: "90564318"

2 1

MAL function syntax
by Moritz Bruder 09 Jan '17

09 Jan '17

Hi, I'm having problems with the exact syntax of MAL functions. I used the syntax from the documentation, which is working for some functions, but my hand-written example doesn't work (neither when pasted in the REPL, nor as input file): "^<identifier> expected" at the end of every line. Any idea what could cause this? (I'm using a fairly recent development version.) Regards, Moritz

2 1

C UDF
by imad hajj chahine 06 Jan '17

06 Jan '17

Hi, After reviewing all the other alternatives like SQL and Python UDF, I was either stuck on performance with SQL UDF or on usability with Python UDF (unable to use with aggregation, and not such great performance with dates), so I decided to go the hard way with C functions, as a bonus it will give me the possibility to change the functionalities without worrying about dependencies, which was not the case in other languages. The purpose is to create a set of formatting functions for Year, Quarter, Month, Week and Day brackets, and of course i need to create the bulk version of each function for performance. Starting from the MTIMEdate_extract_year_bulk, now i have the simple function working, and successfully calling it from mclient: *str* *UDFyearbracket(str *ret, const date *v)* *{* * if (*v == date_nil) {* * *ret = GDKstrdup(str_nil);* * } else {* * int year;* * fromdate(*v, NULL, NULL, &year);* * *ret = (str) GDKmalloc(15);* * sprintf(*ret, "%d", year);* * }* * return MAL_SUCCEED;* *}* For the bulk version i get an error in the log: gdk_atoms.c:1345: strPut: Assertion `(v[i] & 0x80) == 0' failed. *str* *UDFBATyearbracket(bat *ret, const bat *bid)* *{* * BAT *b, *bn;* * BUN i,n;* * str *y;* * const date *t;* * if ((b = BATdescriptor(*bid)) == NULL)* * throw(MAL, "UDF.BATyearbracket", "Cannot access descriptor");* * n = BATcount(b);* * bn = COLnew(b->hseqbase, TYPE_str, BATcount(b), TRANSIENT);* * if (bn == NULL) {* * BBPunfix(b->batCacheid);* * throw(MAL, "UDF.BATyearbracket", "memory allocation failure");* * }* * bn->tnonil = 1;* * bn->tnil = 0;* * t = (const date *) Tloc(b, 0);* * y = (str *) Tloc(bn, 0);* * for (i = 0; i < n; i++) {* * if (*t == date_nil) {* * *y = GDKstrdup(str_nil);* * } else* * UDFyearbracket(y, t);* * if (strcmp(*y, str_nil) == 0) {* * bn->tnonil = 0;* * bn->tnil = 1;* * }* * y++;* * t++;* * }* * BATsetcount(bn, (BUN) (y - (str *) Tloc(bn, 0)));* * bn->tsorted = BATcount(bn)<2;* * bn->trevsorted = BATcount(bn)<2;* * BBPkeepref(*ret = bn->batCacheid);* * BBPunfix(b->batCacheid);* * return MAL_SUCCEED;* *}* PS: I am not a c expert but i can find my way with basic operations and pointers. Any help or suggestions is appreciated. Thank you.

4 17

Upgraded to lastest MonetDB, now get error
by Bryan Senseman 04 Jan '17

04 Jan '17

Ok, I upgraded my Windows version of MonetDB to Dec2016 from Jun2016-SP1 and I get this error. !FATAL: BBPinit: head column must be VOID (ID = 3). I was able to successfully downgrade (uninstall/reinstall) Jun2016-SP1; do I need to install SP2 first? -- ------------------------------ This transmission is confidential and intended solely for the use of the recipient named above. It may contain confidential, proprietary, or legally privileged information. If you are not the intended recipient, you are hereby notified that any unauthorized review, use, disclosure or distribution is strictly prohibited. If you have received this transmission in error, please contact the sender by reply e-mail and delete the original transmission and all copies from your system.

1 0