- users-list - monetdb.org

sql query failure question
by Lynn Carol Johnson 09 Feb '16

09 Feb '16

Hi all - I’m going nuts trying to determine why a monetdb SQL query does not work. I am trying to pull data from 2 tables. These tables are in the DB as shown here: sql>\d TABLE maizefullgenome.fullgenomegff TABLE maizefullgenome.fullgenomestable TABLE maizefullgenome.gene_from_gff TABLE maizefullgenome.ranges_from_gff sql> If I join only on the “pos” field, the query executes. Here is that query: > delChrPosGene_fullGenome <-dbGetQuery(conn, "SELECT fg.chr,fg.pos,gt.gene_name FROM maizeFullGenome.fullgenomestable fg join maizeFullGenome.gene_from_gff gt on (fg.pos >= gt.start_pos AND fg.pos <= gt.end_pos) where gerp_score> 2 AND gerp_neutral_tree_length > 2 AND ((sift_score_a < 0.05 AND sift_score_a >= -1) OR (sift_score_c < 0.05 AND sift_score_c >=-1) OR (sift_score_g < 0.05 AND sift_score_g >=-1) OR (sift_score_t < 0.05 AND sift_score_t >=-1))") If I add the “chr” field, it fails with the message below. Can anyone tell me what is wrong with this second query? > delChrPosGene_fullGenome <-dbGetQuery(conn, "SELECT fg.chr,fg.pos,gt.gene_name FROM maizeFullGenome.fullgenomestable fg join maizeFullGenome.gene_from_gff gt on (fg.chr = gt.chr AND fg.pos >= gt.start_pos AND fg.pos <= gt.end_pos) where gerp_score> 2 AND gerp_neutral_tree_length > 2 AND ((sift_score_a < 0.05 AND sift_score_a >= -1) OR (sift_score_c < 0.05 AND sift_score_c >=-1) OR (sift_score_g < 0.05 AND sift_score_g >=-1) OR (sift_score_t < 0.05 AND sift_score_t >=-1))") Error in .local(conn, statement, ...) : Unable to execute statement 'SELECT fg.chr,fg.pos,gt.gene_name FROM maizeFullGenome.fullgenomestable fg join maizeFullGenome.gene...'. Server says '!Object not found'. Here are the fields in the gene_from_gff table: sql>\d maizefullgenome.gene_from_gff CREATE TABLE "maizefullgenome"."gene_from_gff" ( "chr" INTEGER, "start_pos" INTEGER, "end_pos" INTEGER, "gene_name" CHARACTER LARGE OBJECT ); sql> Here are the fields in the fullgenomestable table: sql>\d maizefullgenome.fullgenomestable CREATE TABLE "maizefullgenome"."fullgenomestable" ( "chr" INTEGER, "pos" INTEGER, "ref_allele" TINYINT, "two_bp_upstream" INTEGER, "two_bp_downstream" INTEGER, "three_bp_upstream" INTEGER, "three_bp_center" INTEGER, "three_bp_downstream" INTEGER, "four_bp_upstream" INTEGER, "four_bp_downstream" INTEGER, "five_bp_upstream" INTEGER, "five_bp_center" INTEGER, "five_bp_downstream" INTEGER, "gc_content" TINYINT, "gerp_neutral_tree_length" REAL, "gerp_score" REAL, "gerp_conserved" BOOLEAN, "cmethylation" REAL, "cpg_methylation_1bk_moving_average" REAL, "chg_methylation_1bk_moving_average" REAL, "chh_methylation_1bk_moving_average" REAL, "mnase_low_minus_high_rpm_shoots" REAL, "mnase_bayes_factor_shoots" REAL, "mnase_hotspot_shoots" BOOLEAN, "mnase_low_minus_high_rpm_roots" REAL, "mnase_bayes_factor_roots" REAL, "mnase_hotspot_roots" BOOLEAN, "allele_total_depth" INTEGER, "site_presence" INTEGER, "conservedgene_presence" BOOLEAN, "minor_allele" INTEGER, "maf" REAL, "combined_site_uniqueness_score" REAL, "site_uniqueness_score_sd" REAL, "expected_invariant_site" BOOLEAN, "codon_position" TINYINT, "within_cds" BOOLEAN, "within_gene" BOOLEAN, "within_transcript" BOOLEAN, "within_exon" BOOLEAN, "within_five_prime_utr" BOOLEAN, "within_three_prime_utr" BOOLEAN, "within_mirna" BOOLEAN, "line_family" INTEGER, "sine_family" INTEGER, "ltr_family" INTEGER, "helitron" BOOLEAN, "carot_common_support_interval" TINYINT, "toco_common_support_interval" TINYINT, "orig_amino_acid_a" TINYINT, "new_amino_acid_a" TINYINT, "amino_acid_substitute_pos_a" INTEGER, "sift_score_a" REAL, "sift_median_seq_info_a" REAL, "number_seqs_at_pos_a" INTEGER, "orig_amino_acid_c" TINYINT, "new_amino_acid_c" TINYINT, "amino_acid_substitute_pos_c" INTEGER, "sift_score_c" REAL, "sift_median_seq_info_c" REAL, "number_seqs_at_pos_c" INTEGER, "orig_amino_acid_g" TINYINT, "new_amino_acid_g" TINYINT, "amino_acid_substitute_pos_g" INTEGER, "sift_score_g" REAL, "sift_median_seq_info_g" REAL, "number_seqs_at_pos_g" INTEGER, "orig_amino_acid_t" TINYINT, "new_amino_acid_t" TINYINT, "amino_acid_substitute_pos_t" INTEGER, "sift_score_t" REAL, "sift_median_seq_info_t" REAL, "number_seqs_at_pos_t" INTEGER, "b73_uniqueness_score" REAL, "cml247_uniqueness_score" REAL, "ph207_uniqueness_score" REAL, "w22_uniqueness_score" REAL, "coelorachis_anc" TINYINT, "vossia_anc" TINYINT, "sorghum_anc" TINYINT, "oryza_anc" TINYINT, "setaria_anc" TINYINT, "brachypodium_anc" TINYINT, "hordeum_anc" TINYINT, "musa_anc" TINYINT, "populus_anc" TINYINT, "vitis_anc" TINYINT, "arabidopsis_anc" TINYINT, "panicum_anc" TINYINT ); sql> Any hints are greatly appreciated! Thanks - Lynn

3 6

WAL read files
by Pierre-Adrien Coustillas 08 Feb '16

08 Feb '16

Hi, is it possible to read WAL files ? sql/ log.<transaction id> why ? We still have a problem of duplicate table . All our monetsdb are updated with the latest version of the repository debian8 , and that several server. We have learned to remove duplicate tables and a stable production. But I want to understand this bug Duplication of tables are always created after a restart monetdb After reading this page https://www.monetdb.org/blog/monetdb-sql-transaction-management-scheme, and particularly this ""By default, persisted transactions are cleaned up from the WAL. During database startup, all transaction still left in the WAL are read and any that are not yet applied are written into the persistent BAT storage"" . I think that the drop or create commands are not cleaned up from the WAL We used the library php _ map.inci and we run a lot of command drop / create ( ... I know this is not recommended ... ) We used autocommit. If I can read and understand what is in the WAL file, can help me understand this bug Thank Pierre -- 1G6 52 route de bischwiller 67300 Schiltigheim Société de Services et de Formations en Logiciels Libres http://1g6.biz Tél : 06 64 63 70 35 // 09 54 38 70 90

3 6

Multi-core join performance
by Koliousis, Alexandros 01 Feb '16

01 Feb '16

Hi, Thank you for your reply. The system has 16 physical cores (2 sockets, 8 cores/sockets, 20MB L3 cache, 256KB L2) and 64GB of RAM. The MonetDB server uses 16 threads to serve my test database and it finds ~63GB of available memory. I used synthetic data to populate the two tables (with names A and B). The particular query I reported about in my previous e-mail generates a lot of output: approx. 500 million tuples. The join predicate is (A.a1 < B.b1) where columns A.a1 and B.b1 contain all unique integers from 1 to 32768. With a lower selectivity (say, 1%, by controlling the values inserted into A.a1 and B.b1), the query executes faster but its behaviour is the same: only 1 thread is utilised. Attached is a trace of the query. Hope this helps, Alexandros PS: Apologies for the separate thread; somehow I ended up subscribing in digest mode > Hi > > This is insufficient information to shed light on it. > Crucial information is system characteristics. > And also a TRACE of your query can shed light on the issue. > > Furthermore, is it a cold or hot execution of the query? > > regards, Martin > > On 20/01/16 13:24, Alexandros Koliousis wrote: > > Hi, > > > > I am joining two tables, each with 32K tuples. The query runs for > ~23-24 seconds on my (multi-core) machine. > > > > During execution, however, only 1 core is utilised, while the rest > of the CPU cores are idle. The cores are under-utilised although I > have configured the MonetDB server to use all the physical cores of my > machine (either by setting 'nthreads' or, in the > > case of mserver5, 'gdk_nr_threads'). > > > > I am running MonetDB server v11.21.13 "Jul2015-SP2", compiled from > source. > > > > Is this due to some configuration setting I missed? Or, due to the > fact that I am running a single-operator query? > > > > Alternatively, I was thinking of creating merge tables, thus > partitioning my tables manually, hoping that this will allow for > intra-operator parallelism. > > > > Thanks, > > > > Alexandros > > > > > > > > _______________________________________________ > > users-list mailing list > > users-list at monetdb.org > > https://www.monetdb.org/mailman/listinfo/users-list > >

3 2

Stored procedure
by poornima＠apakau.com 30 Jan '16

30 Jan '16

Hi, Can you help me with creating a stored procedure in monet ? I keep getting unexpected END even for a simple procedure as follows. create procedure foobar() begin end; I have even tried with the delimiter like in mysql. Nothing works. Regards, Poornima.

2 1

MonetDB test client application not linking
by Ivan Zuboff 30 Jan '16

30 Jan '16

I tried to compile test application ( https://www.monetdb.org/Documentation/Manuals/SQLreference/Programming/MAPI) for MonetDB under Ubuntu 14.04 LTS Trusty Tahr. I followed the download instructions from official site (https://www.monetdb.org/downloads/deb) installation was successful, then I installed a bunch of other packages in order to copmpile it. Now when I try to compile and link test application I get the following errors: libtool: compile: gcc -c -I/usr/include/monetdb test.c -fPIC -DPIC -o .libs/test.o libtool: compile: gcc -c -I/usr/include/monetdb test.c -o test.o >/dev/null 2>&1 libtool: link: gcc -o test test.o -lmapi -lstream -lssl -lcrypto -lcurl -lz test.o: In function `die': test.c:(.text+0x1c): undefined reference to `mapi_explain_query' test.c:(.text+0x27): undefined reference to `mapi_result_error' test.c:(.text+0x3f): undefined reference to `mapi_explain_result' test.c:(.text+0x4a): undefined reference to `mapi_next_result' test.c:(.text+0x5a): undefined reference to `mapi_close_handle' test.c:(.text+0x65): undefined reference to `mapi_destroy' test.c:(.text+0x81): undefined reference to `mapi_explain' test.c:(.text+0x8c): undefined reference to `mapi_destroy' test.o: In function `query': test.c:(.text+0xde): undefined reference to `mapi_query' test.c:(.text+0xf2): undefined reference to `mapi_error' test.o: In function `update': test.c:(.text+0x133): undefined reference to `mapi_close_handle' test.o: In function `main': test.c:(.text+0x190): undefined reference to `mapi_connect' test.c:(.text+0x1a0): undefined reference to `mapi_error' test.c:(.text+0x222): undefined reference to `mapi_fetch_field' test.c:(.text+0x23a): undefined reference to `mapi_fetch_field' test.c:(.text+0x266): undefined reference to `mapi_fetch_row' test.c:(.text+0x276): undefined reference to `mapi_close_handle' test.c:(.text+0x282): undefined reference to `mapi_destroy' collect2: error: ld returned 1 exit status ./t.sh: 11: ./t.sh: ./test: not found What I'm doing wrong? What should I additionally install or change in compilation options for successful linking of test example? Best regards and thanks in advance, Ivan

1 0

A test email (please ignore)
by Ying Zhang 29 Jan '16

29 Jan '16

Dear all, This is just a test e-mail to see if I can still send/receive e-mails to/from this mailing list, since the last e-mail I received here was on Jan. 22nd, 2016. Is it just because MonetDB has been issues free the past days, or is something wrong with the email server? Sorry for bothering you. Regards, Jennie

3 2

Issue with Perl monetdb DBD plugin for DBI on a VM
by Puthick Hok 22 Jan '16

22 Jan '16

Hi, I'm using MonetDB-11.19.7 compiled from source on openSUSE 12.1. I'm also using the Perl client code from MonetDB source to connect to MonetDB from Perl via Perl DBI module. It works fine for a physical server but a VM server with a vmdk drive. It has an issue that the database handler must be disconnected and reconnected in order to execute more SQL statements; otherwise there will be an error saying 'Statement handle not marked as Active'. So I have written 2 testing Perl scripts doing SQL select for 100 iterations. One script is to demonstrate that it will fail while the other will work for all the iterations. The one that works is the script that connect and disconnect to MonetDB in every iteration before executing the SQL. In order to help you replicate the problem, there is an attached tar.gz file containing the 2 perl scripts - one for success and the other one for failure. Due to the size limitation, I cannot include the database dump file for the demonstration. Regards, Puthick

2 1

Query timeout
by Luciano Sasso 21 Jan '16

21 Jan '16

Any idea, how to set query timeout ? Other question, the limit of 65 connection is for merogivian process or mserver ? Att, -- Luciano Sasso Vieira Data Scientist & Solutions Architect luciano(a)gsgroup.com.br <http://www.gsgroup.com.br> | tel: 17 3353-0833 | cel: 17 99706-9335 www.gsgroup.com.br <http://www.gsgroup.com.br> --- Este email foi escaneado pelo Avast antivírus. https://www.avast.com/antivirus

1 0

How to subtract two columns from two different datasets?
by integrity 20 Jan '16

20 Jan '16

Dear experts, I have columns coming from two different datasets which also resides in two different tables in the same layout,now i need the difference of that columns. Example: sql>select (select mag from one_to_many o, targets16 t where o.new_targetid=t.id) - (select mag from one_to_many o,targets16 t where o.old_uniqueid=t.id) ; cardinality violation (22>1) p.s. these two tables have both 22 rows. But if I subtract on single values, it works: select (select avg(mag) from mag1) - (select avg(mag) from mag2); +--------------------------+ | sql_sub_L1 | +==========================+ | -0.015908087452235264 | +--------------------------+ 1 tuple (4.486ms) can any one help me out? Thanks In Advance !!!! Meng

2 1

Multi-core join performance
by Alexandros Koliousis 20 Jan '16

20 Jan '16

Hi, I am joining two tables, each with 32K tuples. The query runs for ~23-24 seconds on my (multi-core) machine. During execution, however, only 1 core is utilised, while the rest of the CPU cores are idle. The cores are under-utilised although I have configured the MonetDB server to use all the physical cores of my machine (either by setting 'nthreads' or, in the case of mserver5, 'gdk_nr_threads'). I am running MonetDB server v11.21.13 "Jul2015-SP2", compiled from source. Is this due to some configuration setting I missed? Or, due to the fact that I am running a single-operator query? Alternatively, I was thinking of creating merge tables, thus partitioning my tables manually, hoping that this will allow for intra-operator parallelism. Thanks, Alexandros

2 1

create table uppercase with " "
by Pierre-Adrien Coustillas 20 Jan '16

20 Jan '16

Hello, bug or feature ? * PETER in uppercase : sql>CREATE TABLE "PETER" ( nom varchar(20) ); operation successful (2.545ms) sql>DROP TABLE PETER; DROP TABLE: no such table 'peter' sql>DROP TABLE "PETER"; operation successful (1.947ms) * peter in lowercase : sql>CREATE TABLE "peter" ( nom varchar(20) ); operation successful (2.917ms) sql>DROP TABLE peter; operation successful (1.882ms) * peter in lowercase and uppercase : sql>CREATE TABLE "peter" ( nom varchar(20) ); operation successful (2.900ms) sql>CREATE TABLE PETER ( nom varchar(20) ); CREATE TABLE: name 'peter' already in use sql>CREATE TABLE "PETER" ( nom varchar(20) ); operation successful (9.339ms) Conclusion : PETER is convert in peter "PETER" is not convert Pierre Coustillas -- 1G6 52 route de bischwiller 67300 Schiltigheim Société de Services et de Formations en Logiciels Libres http://1g6.biz Tél : 06 64 63 70 35 // 09 54 38 70 90

2 1

best practice use schema
by Pierre-Adrien Coustillas 20 Jan '16

20 Jan '16

Hello, What is the best practice on the use of schema ? S hould we always create a schema , or is can I use the " sys" schema by default ? Pierre Coustillas -- 1G6 52 route de bischwiller 67300 Schiltigheim Société de Services et de Formations en Logiciels Libres http://1g6.biz Tél : 06 64 63 70 35 // 09 54 38 70 90

1 0

creating a binary file for "COPY binary ..."
by Lynn Carol Johnson 19 Jan '16

19 Jan '16

Hi all - I'm creating a large db table with several columns. To speed it up, I'm using monetdb's "COPY BINARY" command. I have a question on creating the binary files. What I've done is use java's DataOutputStream to create the binary file - using code that looks as below (sometimes an int, sometimes a real or byte): writerChr = new DataOutputStream(new BufferedOutputStream(new FileOutputStream(chrFile))); writerChr.writeInt(intChrom); When I read this file using a test program and java's DataInputStream, it correctly shows me the integer values I've written. In the "writerChr" case, all values in this particular column are "10". However, when I load this into monetdb the value is way off - it shows a large number. The command I am using to copy this file and one other into a table with 2 "int" columns is: COPY binary into justchrpos from ('/home/lcj34/chrFileC10.bin','/home/lcj34/posFileC10.bin'); It copies in fine, but when I cheek the values I see this: (not 10!) sql>\d justchrpos CREATE TABLE "testbig"."justchrpos" ( "chr" INTEGER, "pos" INTEGER ); sql>select * from justchrpos limit 10; +-----------+-----------+ | chr | pos | +===========+===========+ | 167772160 | 16777216 | | 167772160 | 33554432 | | 167772160 | 50331648 | | 167772160 | 67108864 | | 167772160 | 83886080 | | 167772160 | 100663296 | | 167772160 | 117440512 | | 167772160 | 134217728 | | 167772160 | 150994944 | | 167772160 | 167772160 | +-----------+-----------+ 10 tuples (3.601ms) sql> QUESTION: Is using java's DataOutputStream a valid way to create the binary files for monetdb? If not, what should I be using? As I mentioned above, when I read back the created file using java's DataInputStream, I see the values "10" as I expect. Thanks - Lynn

3 3

Monet way of handling string columns
by bhuvan Mitra 18 Jan '16

18 Jan '16

Hello, I have recently started looking into Monetdb and want to contribute to it. My complete focus is on string columns as numeric data can already be processed with maximum speed. I know it uses BATs to store every single column, but I read some where that it uses dictionary for string columns. How is it actually using dictionary? And are there any other special handling techniques for strings? When a query like 'contains' comes, how will monet handle that? I found there is no inverted index in monet to deal queries like this. (Please tell a rough model of how query execution is done incase of string columns) Hope this forum gives me some insight of what's happening during string column processing. Thanks in advance, -Mitra

1 0

MonetDB-11.21.13 (Jul2015-SP2) can't start in CentOS 6.3
by chenkaijiang＠bigknow.com.cn 18 Jan '16

18 Jan '16

I downloaded the source codes at https://www.monetdb.org/downloads/sources/Jul2015-SP2/MonetDB-11.21… ./configure --prefix=/usr/local/MonetDBmakemake install No error is reported. But can't start:monetdbd create ./ycmonetdbd: error while loading shared libraries: libstream.so.7: cannot open shared object file: No such file or directory It looks like the libstream.so is lib on debian linux.  How can I fix it on CentOS 6.3?

2 1

How to set query timeout in MonetDB Server
by Luciano Sasso 16 Jan '16

16 Jan '16

How to set query timeout in MonetDB Server? -- Luciano Sasso Vieira Data Scientist & Solutions Architect luciano(a)gsgroup.com.br <http://www.gsgroup.com.br> | tel: 17 3353-0833 | cel: 17 99706-9335 www.gsgroup.com.br <http://www.gsgroup.com.br> --- Este email foi escaneado pelo Avast antivírus. https://www.avast.com/antivirus

1 0

Announcement: New Jul2015-SP2 Bugfix release of MonetDB suite
by Sjoerd Mullender 16 Jan '16

16 Jan '16

The MonetDB team at CWI/MonetDB BV is pleased to announce the Jul2015-SP2 bugfix release of the MonetDB suite of programs. More information about MonetDB can be found on our website at <http://www.monetdb.org/>. For details on this release, please see the release notes at <http://www.monetdb.org/Downloads/ReleaseNotes>. As usual, the download location is <http://dev.monetdb.org/downloads/>. Jul 2015-SP2 bugfix release Bug Fixes * 2014: 'null' from copy into gets wrong * 3817: opt_pushselect stuck with multi-table UDF * 3835: windows does not release ram after operations * 3836: rand() only gets evaluated once when used as an expression * 3838: Update column with or without parenthesis produce different results * 3840: savepoints may crash the database * 3841: mclient fails with response "Challenge string is not valid" * 3842: SQL execution fails to finish and reports bogus error messages * 3845: Too many VALUES in INSERT freeze mserver5 * 3847: Wrong SQL results for a certain combination of GROUP BY / ORDER BY / LIMIT * 3848: mserver segfault during bulk loading/updating * 3849: HUGEINT incorrect value * 3850: DEL character not escaped * 3851: expression that should evaluate to FALSE evaluates to TRUE in SELECT query * 3852: CASE statement produces GDK error on multithreaded database: BATproject does not match always * 3854: Complex expression with comparison evaluates incorrectly in WHERE clause * 3855: Foreign key referencing table in a different schema - not allowed. * 3857: Large LIMIT in SELECT may abort the query * 3861: Using window functions cause a crash * 3864: Error in bulk import for chinese character * 3871: NOT x LIKE triggers "too many nested operators" * 3872: mserver crashes under specific combination of JOIN and WHERE conditions * 3873: mserver5: gdk_bat.c:1015: setcolprops: Assertion `x != ((void *)0) || col->type == 0' failed. * 3879: Database crashes when querying with several UNION ALLs. * 3887: Querying "sys"."tracelog" causes assertion violation and crash of mserver5 process * 3889: read only does not protect empty tables * 3895: read only does not protect this table

2 1

error opening DB on External drive
by Lynn Carol Johnson 15 Jan '16

15 Jan '16

Hello - I am running monetdb with the July2015 service pack on MAC OSX. I have no problem running and connecting to a dbfarm that lives on my MAC. But I want to test a very large data set so have created a dbfarm on a Samsung T1 SSD external drive that is USB connected to my MAC. I have successfully started monetdbd server and have attached to my dbfarm. I created/released a new database. Yet when I attempt to run mClient it fails. Here is the "get all" command: rs-btlcj34lt:testMapiSock lcj34$ monetdbd get all /Volumes/Samsung_T1/alt-dbfarm property value hostname rs-btlcj34lt.maize.cornell.edu dbfarm /Volumes/Samsung_T1/alt-dbfarm status monetdbd[740] 1.7 (Jul2015-SP1) is serving this dbfarm mserver /Users/lcj34/development/usr/local/monetdb/bin/mserver5 logfile /Volumes/Samsung_T1/alt-dbfarm/merovingian.log pidfile /Volumes/Samsung_T1/alt-dbfarm/merovingian.pid sockdir /tmp port 50000 exittimeout 60 forward proxy discovery true discoveryttl 600 control no passphrase <unknown> mapisock /tmp/.s.monetdb.50000 controlsock /tmp/.s.merovingian.50000 rs-btlcj34lt:testMapiSock lcj34$ Here is the command line error: rs-btlcj34lt:alt-dbfarm lcj34$ mclient -u monetdb -d maizeFullGenomeDB password: monetdbd: internal error while starting mserver, please refer to the logs And here is the error from the log file. 2016-01-14 10:33:26 MSG maizeFullGenomeDB[749]: # Visit http://www.monetdb.org/ for further information 2016-01-14 10:33:26 MSG maizeFullGenomeDB[749]: !IOException:mal_mapi.listen:operation failed: binding to UNIX socket file /Volumes/Samsung_T1/alt-dbfarm/maizeFullGenomeDB/.mapi.sock failed: No such file or directory If I switch back to a dbfarm local to the MAC, it works fine. Any ideas why I can't connect to a dbfarm on the external drive? Is there something else that needs to be setup for the socket creation to work? Thanks - Lynn

3 3

Error on database startup
by Robin Cijvat 13 Jan '16

13 Jan '16

Hello everyone! Every now and then my laptop fails to wake up from standby mode and I have to do a hard reset, while monetdb is running in the background. I have had the following problem multiple times lately when trying to start the database that was stopped during the hard reset: monetdb start tpch starting database 'tpch'... FAILED start: starting 'tpch' failed: database 'tpch' did not initialise the sql scenario This is from the log: 2016-01-13 07:22:38 MSG tpch[7085]: # Start processing logs sql/sql_logs version 52200 2016-01-13 07:22:38 ERR tpch[7085]: !mvc_init: unable to create system tables 2016-01-13 07:22:38 MSG tpch[7085]: !SQLException:SQLinit:Catalogue initialization failed 2016-01-13 07:22:38 MSG tpch[7085]: !ERROR: Could not read the version number from the file 'sql_logs/sql//log'. 2016-01-13 07:22:38 MSG merovingian[7066]: database 'tpch' (7085) has exited with exit status 0 2016-01-13 07:22:38 MSG merovingian[7066]: database 'tpch' has shut down 2016-01-13 07:22:38 ERR control[7066]: (local): failed to fork mserver: database 'tpch' did not initialise the sql scenario Now I don't keep any important information in this database, so for me it is fine to reload everything. However, since this is a recurring problem, I wondered if anyone has any suggestions as to how to fix such a 'broken' database.

1 0

New syntax filter functions
by Roberto Cornacchia 08 Jan '16

08 Jan '16

Hi there, I'm a bit struggling with the syntax for filter function, after it changed from Oct2014 to Jul2015. I have looked at the release notes, but they don't seem to even mention this change of syntax. This used to work in Oct2014: create filter function maxlevenshtein(s1 string, s2 string, k int) external name str."distjoin_levenshtein"; select t.name, f.name from sys.tables t, sys.functions f where t.name maxlevenshtein(2) f.name; In Jul2015, if I look at Niels' comment in https://www.monetdb.org/bugzilla/show_bug.cgi?id=3702 , it should be something similar to select t.name, f.name from sys.tables t, sys.functions f where [t.name] maxlevenshtein [f.name, 2]; But this is still not right. How should it be? And should I change the SQL / MAL / C interface of the function as well? Thanks,

1 2

Monitoring MonetDB performance
by Alexander Gordt 07 Jan '16

07 Jan '16

Hello together, I’ve got a performance issue with my project, where the DB performance is going down for what ever reason. The activity monitor of OSX shows me the following numbers about the mserver5-process: The CPU usage is very low (15-25% of one core!), the IO traffic on my ssd is fair (about 10-40 MB/s), but the speed slows down. I’ve had a look in the merovingian.log, but there’s only the connection shown. For the first 5000 messages, the CPU usage of mserver5 was about 70-110% - why is it going down?… How could I analyze, why the database performance is going down? Is it possible, that there is an issue with the JDBC-driver, that the performance is decreasing? I attache a log file where you can see that the performance is quite good for some time, but then goes down for whatever reason. 18:40:46 [main] INFO ModelPersistenceTest - Finished BatchInsert of 200 message in 15.711 seconds - totally inserted 5200 messages 18:40:46 [main] INFO ModelPersistenceTest - Starting Batch-Insert for of 200 message 18:41:03 [main] INFO ModelPersistenceTest - Finished BatchInsert of 200 message in 17.189 seconds - totally inserted 5400 messages 18:41:03 [main] INFO ModelPersistenceTest - Starting Batch-Insert for of 200 message 18:41:27 [main] INFO ModelPersistenceTest - Finished BatchInsert of 200 message in 23.834 seconds - totally inserted 5600 messages 18:41:27 [main] INFO ModelPersistenceTest - Starting Batch-Insert for of 200 message 18:42:19 [main] INFO ModelPersistenceTest - Finished BatchInsert of 200 message in 52.13 seconds - totally inserted 5800 messages 18:42:19 [main] INFO ModelPersistenceTest - Starting Batch-Insert for of 200 message 18:43:09 [main] INFO ModelPersistenceTest - Finished BatchInsert of 200 message in 49.892 seconds - totally inserted 6000 messages 18:43:09 [main] INFO ModelPersistenceTest - Starting Batch-Insert for of 200 message 18:43:59 [main] INFO ModelPersistenceTest - Finished BatchInsert of 200 message in 49.786 seconds - totally inserted 6200 messages 18:43:59 [main] INFO ModelPersistenceTest - Starting Batch-Insert for of 200 message 18:45:31 [main] INFO ModelPersistenceTest - Finished BatchInsert of 200 message in 92.389 seconds - totally inserted 6400 messages 18:45:31 [main] INFO ModelPersistenceTest - Starting Batch-Insert for of 200 message 18:47:26 [main] INFO ModelPersistenceTest - Finished BatchInsert of 200 message in 114.412 seconds - totally inserted 6600 messages 18:47:26 [main] INFO ModelPersistenceTest - Starting Batch-Insert for of 200 message 18:49:24 [main] INFO ModelPersistenceTest - Finished BatchInsert of 200 message in 118.324 seconds - totally inserted 6800 messages One message results in 10-50 insert statements in different tables and the 200 messages are processed in one transaction. The absolute performance isn’t nice, but that is another topic – I’ve read about "copy into“ and that kind of stuff – but that gonna be part in a later optimization. I tried to find out, if there is a long running process, but „select * from sys.queue;“ didn’t show me anything of interest – the result shows only this select. Is there a way to gather some statistics about executed queries and their runtime? Or about blocking resources… Thanks in advance for bringing any light in the black box :) Best regards, Alex

1 1

About JDBC Driver
by Luciano Sasso 07 Jan '16

07 Jan '16

Steps for error: 1- In testing with java client and JDBC 2:19, an error occurs when saving decimal field, for example the value 11:23 is recorded as 13. 2- When I run the same test with the 2.8 driver, saves normally and correct. Not only is the 2.19 driver, all above 2.8 is that same mistake. My Java client is Pentaho Data Integration, My MonetDB is JUL2015-SPI I go zip package test, and send fo tests. Att, -- Luciano Sasso Vieira Data Scientist & Solutions Architect luciano(a)gsgroup.com.br <http://www.gsgroup.com.br> | tel: 17 3353-0833 | cel: 17 99706-9335 www.gsgroup.com.br <http://www.gsgroup.com.br> --- Este email foi escaneado pelo Avast antivírus. https://www.avast.com/antivirus

2 1

About database crack
by chenkaijiang＠bigknow.com.cn 06 Jan '16

06 Jan '16

Hi, all, I find that in Oct2010 branch (http://dev.monetdb.org/hg/MonetDB/shortlog/Oct2010) it looks like that the database cracking technology has been implemented. But looks like that it wasn't merged into the latest release (Jul2015), right? Why not merge this into the latest release? Is there any plan to merge it in the future?

3 3

MonetDBLite error in R
by Iain Wallace 06 Jan '16

06 Jan '16

hi all, I was wondering if anybody could suggest how to debug this error: Server says 'GDKerror:!ERROR: BATsetaccess: tmp_3505 has -1 views I am trying to do a simple join using dplyr sample%>%left_join(interaction_pairs) The tables appear to be working correctly as I can collect them and use them in R properly interaction_pairs<-collect(interaction_pairs) Thanks Iain

2 1

invalid credentials using mserver5
by Xu,Wenjian 06 Jan '16

06 Jan '16

Hi all, Normally I am using *monetdbd*+"mclient" combination to use MonetDB. Now, since I want to see the debug info in each gdk operator, I have to use *mserver5*. I do as follows (JULY2015-SP1+ubuntu14.04): Step1: $ mserver5 --dbpath=${HOME}/[my_dbpath] --daemon=yes --algorithms Step2: In another console: $ mclient -d [dbname] -u monetdb password: monetdb It replies: InvalidCredentialsException:checkCredentials:invalid credentials for user 'monetdb' However, user:monetdb, pwd:monetdb is the default one and I can use it successfully in *monetdbd*+"mclient" combination. Any idea? Thanks, XU Wenjian

2 1