- users-list - monetdb.org

MonetDb Feedback/Memory question
by Eduardo Oliveira 04 Nov '12

04 Nov '12

I work for a company Adclick on Portugal, and we are trying MonetDb and benchmarking it. We are doing it at AWS (Amazon Web Services), that IO is very low, so we didn't expect great numbers, but we are very surprised great numbers appear. Now we will benchmark on server with SSD disks. I'm really enjoying MonetDb, *very good work *developing team. I think if MonetDb team work in replication master-slave, on real delete's! more rpm's for CentOS, AMI to Amazon Cloud, MonetDb can become real a top commercial product (i already tried so many). I have a question regarding memory: "MonetDB excessively uses main memory for processing, but does not require that all data fit in the available physical memory. To handle dataset that exceed the available physical memory, MonetDB does not (only) rely on the available swap space, but (also) uses memory-mapped files to exploit disk storage beyond the swap space as virtual memory." "For example, while bulk-loading data (preferably via a COPY INTO statements from a (possibly compressed) CSV file), MonetDB need to have all columns of the table that is currently being loaded "active", i.e., accessable in the address space. However, during loading, parts of the data are continuously written to the persisten files on disk, i.e., the whole table does not have to fit into main memory. E.g., loading a 100 GB table works fine on a system with 8 GB RAM and 16 GB swap -- provided there is sufficient free disk space." Bulk-loading is not a problem since we split CSV in chunks of 2-5M, but i'm more concern about queries, our datasize will be around 100-150GB ~1B rows and we are thinking on 32GB Memory server. I think 32GB memory will be more than enough for any query below: Our queries will be select ..., sum(...) where senddate>X and senddate<Y group by K, [Z] Or select ..., sum(...) where somecolumn = T and senddate>X and senddate<Y group by K, [Z] We will have just one table, and we will have some concurrency, not a lot, we expect 50-200 (depending on the time of day) queries per minute. So it can happen 3-5 queries at the same time. MonetDb will be able to handle this with 32GB of memory without swapping? I watch http://www.youtube.com/watch?v=yrLd-3lnZ58 and in our cause with our queries with senddate>X and senddate<Y it looks like that concurrent queries will not be able to share data between. Of course i will test concurrency before putting on production, but if i can get a rough answer will be great. Thanks a lot in advance. -- *Eduardo Oliveira * *IT* ***Email:* eduardo.oliveira(a)adclick.pt <nuno.morais(a)adclick.pt> *Web: * www.adclickint.com <http://www.adclickint.com/> * * <http://www.adclickint.com/> <http://www.adclickint.com/> _______________________________________________ users-list mailing list users-list(a)monetdb.org http://mail.monetdb.org/mailman/listinfo/users-list

2 1

unexpected IDENT
by Marcelo Lacerda 04 Nov '12

04 Nov '12

Hey there, when I run this create table dim_date (id integer autoincrement primary key, date date unique not null); I get this syntax error, unexpected IDENT, expecting ')' or ',' in: "create table dim_date (id integer autoincrement" Is that expected? I'm running a compiled version of monet 11.13.3 on a debian machine. -- Marcelo Lacerda _______________________________________________ users-list mailing list users-list(a)monetdb.org http://mail.monetdb.org/mailman/listinfo/users-list

2 1

Add, alter, delete user syntax
by Marcelo Lacerda 03 Nov '12

03 Nov '12

Greetings, where can I find the syntax for the database maintenance commands? Is there anything similar to the postgresql documentation[1]? -- Marcelo Lacerda [1] - http://www.postgresql.org/docs/9.2/static/sql-createuser.html _______________________________________________ users-list mailing list users-list(a)monetdb.org http://mail.monetdb.org/mailman/listinfo/users-list

3 2

JDBC bug with limit and offset?
by Percy Wegmann 02 Nov '12

02 Nov '12

I'm running MonetDB on Ubuntu 10.04, with Monet built from the Oct2012_release <http://dev.monetdb.org/hg/MonetDB/rev/3c157ea4be53> tag. I have a client program that connects using the JDBC driver and issues a simple SQL query using limit and offset. I see that: - limit and offset work when using literal parameters in the SQL string - limit works by itself when using a bind parameter - offset works by itself when using a bind parameter - when specifying both limit and offset and using a bind parameter for either or both, the merovingian segfaults with a message like this: database 'click' (2428) was killed by signal SIGSEGV Examples of queries that work: SELECT * FROM the_table LIMIT 5 OFFSET 100; SELECT * FROM the_table LIMIT ?; SELECT * FROM the_table OFFSET ?; Example of queries that don't work: SELECT * FROM the_table LIMIT ? OFFSET ?; SELECT * FROM the_table LIMIT 5 OFFSET ?; SELECT * FROM the_table LIMIT ? OFFSET 100; I've tried passing in the limit and offset as 64 bit integers (long), 32 bit integers (int) and strings, and none of these work. I also tried downgrading to MonetDB Database Server v1.6 (Jul2012-SP2) but had the same issue. Has anyone else run into this? Is this a new bug that I should file? Thanks, Percy Wegmann _______________________________________________ users-list mailing list users-list(a)monetdb.org http://mail.monetdb.org/mailman/listinfo/users-list

2 5

Arrows in the sql command line
by Masood Mortazavi 30 Oct '12

30 Oct '12

For some reason, when I use > mclient -u voc -d voc going into the sql command interface, I am unable to recall earlier sql commands using the arrow keys on the keyboard. Some other friends and colleagues can do this. What gives? Regards, Masood _______________________________________________ users-list mailing list users-list(a)monetdb.org http://mail.monetdb.org/mailman/listinfo/users-list

2 2

persistent MAL function
by Liwen -George- Zhang 30 Oct '12

30 Oct '12

Hi,A user MAL function created in one connection is destroyed when logoff. What's the correct way to persistent it? I finally want to create a SQL function based on this MAL function.For example: msql>function user.malhistogram(db:str,tablename:str, columnname:str):bat[:any_1,:int]; X_2 := sql.mvc(); X_3 := sql.bind(X_2,db,tablename,columnname,0); return aggr.histogram(X_3); end malhistogram; msql>h := malhistogram("voc","voyages","trip"); msql>io.print(h); #-----------------# # h t # name # int int # type #-----------------# [ 1, 3038 ] [ 2, 1907 ] [ 3, 1288 ] [ 4, 852 ] ... thanks a lot. George Zhang _______________________________________________ users-list mailing list users-list(a)monetdb.org http://mail.monetdb.org/mailman/listinfo/users-list

1 0

#GDKrealloc(21474836496) fails when try TCP-H benchmark with scale factor 100 on 64 bit ubuntun
by joe cooper 29 Oct '12

29 Oct '12

Hi, I got #GDKrealloc(21474836496) fails error when I tried to run TPC-H benchmark with a scale factor 100. The error happened during loading tables. Here are information about my configuration: The monetdb version used: changeset: 45737:bfd558546498 tag: tip user: Martin Kersten <mk(a)cwi.nl> date: Thu Oct 25 21:55:38 2012 +0200 summary: Add missing variable. The machine used: 64 bit Ubuntu natty With 7.806 GiB available main-memory. Compiler option used: 64 bit OIDs used Runtime options used: /home/borgdb/monetdb/install/bin/mserver5 --set gdk_dbfarm=/home/borgdb/monetdb/dbfarm --dbname=tpch100 --set merovingian_uri=mapi:monetdb://sc-borgdb6:50000/tpch100 --set mapi_open=false --set mapi_port=0 --set mapi_usock=/home/borgdb/monetdb/dbfarm/tpch100/.mapi.sock --set monet_vault_key=/home/borgdb/monetdb/dbfarm/tpch100/.vaultkey --set gdk_nr_threads=8 --set max_clients=64 --set sql_optimizer=default_pipe --set monet_daemon=yes I am new to monetdb, so I am not sure if that is a known issue. If it is a known issue, my question is: on a 64 bit machine with 7.8GiB memory, how big a scale factor(TPC-H benchmark) can be handled by monetdb with 64 bit OID? If there any runtime option that enables monetdb to handle a big scale factor when run TPC-H benchmark? On my configuration, I succeeded in running TPC-H with scale factor 10, but failed with every scale factor >= 15. Any comments will be greatly appreciated, Thanks, Joe _______________________________________________ users-list mailing list users-list(a)monetdb.org http://mail.monetdb.org/mailman/listinfo/users-list

2 1

Funnel deployment
by Tapomay Dey 25 Oct '12

25 Oct '12

Do the funnel/"monetdb discover" features rely on UDP broadcast. I see there is no option to specify IP address-port while configuring a funnel. Also UDP broadcast is not so conducive to security in networks with large number of nodes. Is there any way discover and funnel features would work if there are VLANs on the switches that prevent UDP discovery. Regards, Tapomay. _______________________________________________ users-list mailing list users-list(a)monetdb.org http://mail.monetdb.org/mailman/listinfo/users-list

2 2

log file not found after a crash
by Medha Atre 23 Oct '12

23 Oct '12

Hi, I am running MonetDB v. 11.9.7. I have a large RDF dataset loaded in it (800+ million triples) as separate predicate tables for each unique predicate. I am running some simple one join SPAQRL queries ("?s :p1 ?o JOIN ?o :p2 ?x" kind) in bulk through a shell script (echo "<some query>" | mclient -drdf -lsql >> results). Sometime overnight when the queries were running, an error occurred in the database run, and it crashed by spitting out messages "monetdbd: an internal error has occurred, refer to the logs for details, please try again later" and "monetdbd: internal error while starting mserver, please refer to the logs" for the rest of the queries after some query caused this error. But when I tried to open "<Path To MonetDB installation>/var/log/monetdb/merovingian.log", it said "no such file or directory". I restarted the server and the particular database. It shows me its status as "health 97%" and "crash <crash date and time>". I tried to find the logfile again, but it's not there. I am wondering how I can know what the problem was. _______________________________________________ users-list mailing list users-list(a)monetdb.org http://mail.monetdb.org/mailman/listinfo/users-list

5 10

Re: PHP Connectivity
by Jeremy O'Sullivan 23 Oct '12

23 Oct '12

Hi Fabian, I changed the version of PHP from 5.4.4 to 5.2.17 and it worked. Thanks, Jeremy _______________________________________________ users-list mailing list users-list(a)monetdb.org http://mail.monetdb.org/mailman/listinfo/users-list

1 0