- users-list - monetdb.org

[MonetDB-users] MonetDb comparison Data Load
by Prometheus WillSurvive 19 Jan '09

19 Jan '09

Dear Developers-Users, First of all I would like to thanks all of your afforts to birng us these DB technologies and a platform for openly discussion. We are consulting big sized customers and deploying their data search technologies. Total size is around 1 petabytes daily . (total of our customers number ) . Their data size starts 1 TB (9.000.000.000 lines cvs) to 100 TB. so we have different size of customer base. We are looking opensource db solutions for a while due to lack of support and technological cumbersome of the current commercial solutions. We have 3 test customer now : 1-1 TB data dailiy read only purposes. 9.000.000.000 cvs file should be load to database . Data should be load to the database max 2 hours. after that comlex crm analitic queries. max 10 concurrent user/queries. database will be daily and wil not grow dailiy. so limit is 1 TB data load. 2-10 TB data daily. same as above .. 3-50 TB data stabil. will not grow. So we are looking for a solutions: *Fast data loading. How fast we load data to the MonetDB ? Is it could be multithreaded data loading possible ? *Is it possible to MPP ? can we add more server to share cpu resources and decrease the load time ? what is the best HW solution for above first customer ? Do we need still fat enterrprise servers or what ? We are also looking for InfoBright, Vertica and also Luciddb. VErtica is not opensource so out of the list. Infobright seems not giving its full power in the opensource.. Also we heard about the MonetDB X100 and interested. After 3 months succefully test we will support monetDB developer if its solved our problems. We can pay . Best Regards W.S

1 0

[MonetDB-users] Are there binary rpm install files for Red Hat Enterprise Linux?
by 小波顾 17 Jan '09

17 Jan '09

_________________________________________________________________ More than messages–check out the rest of the Windows Live™. http://www.microsoft.com/windows/windowslive/

1 0

[MonetDB-users] Copy from file which contains Chinese characters failed.
by 小波顾 17 Jan '09

17 Jan '09

Copy from file which contains Chinese characters failed. My test table is create table t1(a int not null primary key, b varchar(1024)); My test file d:\data.txt contains two lines1,测试12,测试2 I have tried to save the data file using ANSI and UTF8 encoding using Notepad on windows,and tried to start mclient with -Ecp437, -Ecp936 and -Eutf-8 option, neither combination of thefile encoding and mclient encoding can load the file correctlly. When both using utf-8 the following error messages appears:sql>copy into t1 from 'd:\\data.txt' using delimiters ',', '\r\n';!SQLException:importTable:failed to import table!ERROR: value 'ï»?' while parsing 'ï»?,é¡¾å°æ³?!ERROR: 2,è¦ƒæ€? from line 0 field 0 not inserted, expecting type int!ERROR: TABLETload_bulk: read error (after loading 0 records)0 tuples When using other combinations, the following messages appears:sql>copy into t1 from 'd:\\data.txt' using delimiters ',', '\r\n';0 tupleswrite errorsql> The following steps works, but the exported data.txt isn't readable by human. mclient XXXXX -Ecp437 delete from t1;insert into t1 values(1,'²âÊÔ1');insert into t1 values(2,'²âÊÔ2'); copy select * from t1 into 'd:\\data.txt' using delimiters ',' ,'\r\n'; delete from t1;copy into t1 from 'd:\\data.txt' using delimiters ',' ,'\r\n'; _________________________________________________________________ News, entertainment and everything you care about at Live.com. Get it now! http://www.live.com/getstarted.aspx

3 8

[MonetDB-users] Can't connect to remote MonetDB5-SQL Server.
by 小波顾 17 Jan '09

17 Jan '09

I have installed all the rpms in my Fedora9 virtual machine: 2009-01-17 12:07 3,339,229 MonetDB-1.26.2-1.fc9.oid32.i386.rpm2009-01-17 12:07 84,115 MonetDB-client-1.26.2-1.fc9.oid32.i386.rpm2009-01-17 12:37 7,988 MonetDB-SQL-2.26.2-1.fc9.oid32.i386.rpm2009-01-17 12:36 873,636 MonetDB-SQL-server5-2.26.2-1.fc9.oid32.i386.rpm2009-01-14 22:07 5,054,522 MonetDB5-server-5.8.2-1.fc9.oid32.i386.rpmAnd using the following steps recreated the demo database: shell>merovingian ###Start the proxy damo shell>monetdb destroy demo shell>monetdb create demo shell>monetdb start demo Now I can connect to the demo database using mclient -lsql -d demo from the local machine, then I changed "mapi_open=true" in the file /etc/monetdb5.conf and restarted all the server processes, but I still can't connect to databases from remote machines. BTW, how to stop the merovingian process, I used kill:( _________________________________________________________________ Show them the way! Add maps and directions to your party invites. http://www.microsoft.com/windows/windowslive/events.aspx

1 1

[MonetDB-users] SIGBUS on mserver5 after large load.
by Schwarz, Omri 16 Jan '09

16 Jan '09

I have a large database that loads nicely on MonetDB from MySQL when it'd done with INSERTs with one row at a time. But when I group them in sets of 300 or so, for any reason, i.e. INSERT INTO TABLE VALUES (FOO), (BAR) ... ; it develops a memory leak which is not helped by setting gdk_mem_bigsize or gdk_vm_minsize. And when that makes me abort the load, the result is that backend data are corrupted. I can start the database server, but on the first connection attempt, this happens: [pid 18878] read(19, "\0\0", 2) = 2 [pid 18878] close(19) = 0 [pid 18878] stat("bat/02/224.tail", {st_mode=S_IFREG|0600, st_size=254, ...}) = 0 [pid 18878] stat("bat/02/224.tail.new", 0x416d9b00) = -1 ENOENT (No such file or directory) [pid 18878] open("bat/02/224.tail", O_RDONLY) = 19 [pid 18878] read(19, "\1\1\1\1\1\1\1\1\1\1\1\1\1\1\1\1\1\1\1\1\1\1\1\1\1\1\1\1\1\1\1\1"..., 254) = 254 [pid 18878] close(19) = 0 [pid 18878] --- SIGBUS (Bus error) @ 0 (0) --- Process 18878 detached [pid 18877] +++ killed by SIGBUS +++ +++ killed by SIGBUS +++ So, a couple questions: 1. how hard would it be to write a backend integritiy checker for the binary association tables? 2. what more can I do to make sure all the changes in the backend are truly atomic ? 3. Are there good prospects for this memory leak being traced and addressed? 4. Can the BAT file format be changed to have more checks on its integrity? (checksums, sentinel variables, etc) 5. Might it make sense for the backend files to have a format relying on existing database libraries? BDB? Denham Quantitative Management LP 200 Clarendon Street, 25th Floor Boston, MA 02216 T: 617-587-5812 M: 617-909-7786 omri.schwarz(a)denhamcapital.com NOTICE: This e-mail message is intended for the named recipient(s) above and may contain confidential information. If you are not the intended recipient, you are hereby notified that any dissemination, distribution, or copying of this e-mail and any attachment(s) is strictly prohibited. If you have received this e-mail in error, please immediately notify the sender by replying to this e-mail and delete the message and any attachment(s) from your system. NO REPRESENTATION OR WARRANTY IS MADE TO ANY PERSON AS TO THE ACCURACY OR COMPLETENESS OF THE INFORMATION CONTAINED HEREIN. Denham Capital Management(sm) is a trademark of Denham Capital Management LP.

1 0

[MonetDB-users] Python client strangeness.
by Schwarz, Omri 15 Jan '09

15 Jan '09

when you set "gdk_debug" to 4 (for memory debugging), it causes the server to give empty lists to "select * from tables;" queries coming from Python DB API clients, but not when they come from mclient. Denham Quantitative Management LP 200 Clarendon Street, 25th Floor Boston, MA 02216 T: 617-587-5812 M: 617-909-7786 omri.schwarz(a)denhamcapital.com NOTICE: This e-mail message is intended for the named recipient(s) above and may contain confidential information. If you are not the intended recipient, you are hereby notified that any dissemination, distribution, or copying of this e-mail and any attachment(s) is strictly prohibited. If you have received this e-mail in error, please immediately notify the sender by replying to this e-mail and delete the message and any attachment(s) from your system. NO REPRESENTATION OR WARRANTY IS MADE TO ANY PERSON AS TO THE ACCURACY OR COMPLETENESS OF THE INFORMATION CONTAINED HEREIN. Denham Capital Management(sm) is a trademark of Denham Capital Management LP.

1 0

[MonetDB-users] Building clients - php and perl dependencies
by David Svenson 15 Jan '09

15 Jan '09

Hi, I am trying to build MonetDB on windows, and so far I have successfully built 'buildtoos' and 'MonetDB' from the CVS. Now I'm trying to build 'clients' without PHP or PERL, but I'm having difficulties. The documentation says that only Phyton is required, however, I still get build errors related to perl and php dependencies. I do not set HAVE_PERL=1 or HAVE_PHP=1, and therefor expect the build not to rely on them. Am I missing something? Is there something else I need to do in order to build this project? -- View this message in context: http://www.nabble.com/Building-clients---php-and-perl-dependencies-tp214802… Sent from the monetdb-users mailing list archive at Nabble.com.

2 3

[MonetDB-users] MALException:MAL.getAddress:address of 'bat.BKChasReadMode' not found (followup)
by Schwarz, Omri 15 Jan '09

15 Jan '09

I found the problem: the linker was silently balking at creating the .sos and not telling 'make' about it, so a full build was done without the shared libraries. It's a GNU linker/autoconf issue that the MonetDB team can't do much about besides posting this email where Google will find it, but thank you all for helping me diagnose the issue. Now I just have to work with the memory leak I'm seeing. I see I can log all memory operations, but where can I set memory usage to be limited altogether? Denham Quantitative Management LP 200 Clarendon Street, 25th Floor Boston, MA 02216 T: 617-587-5812 M: 617-909-7786 omri.schwarz(a)denhamcapital.com NOTICE: This e-mail message is intended for the named recipient(s) above and may contain confidential information. If you are not the intended recipient, you are hereby notified that any dissemination, distribution, or copying of this e-mail and any attachment(s) is strictly prohibited. If you have received this e-mail in error, please immediately notify the sender by replying to this e-mail and delete the message and any attachment(s) from your system. NO REPRESENTATION OR WARRANTY IS MADE TO ANY PERSON AS TO THE ACCURACY OR COMPLETENESS OF THE INFORMATION CONTAINED HEREIN. Denham Capital Management(sm) is a trademark of Denham Capital Management LP.

1 0

[MonetDB-users] MALException:MAL.getAddress:address of 'bat.BKChasReadMode' not found (followup)
by Schwarz, Omri 14 Jan '09

14 Jan '09

Making some progress. Starting with a fresh merovingian process, I did an "strace -fe file,process -p PID] on it. "monetdb create test" did nothing, but "start test" shows it opens the dbfarm directory, and mal_init.mal, but then fails to find any of the MonetDB5 .sos [excriciating scroll included below] because only the .a and .la files got installed there. The build tree is also stripped of .sos. [root@mork MonetDBFast]# ls lib/MonetDB5/lib lib_aggr.a lib_batifthen.la lib_clients.a lib_inet.la lib_mal_scheduler.a lib_pqueue.la lib_streams.la lib_aggr.la lib_batmath.a lib_clients.la lib_inspect.a lib_mal_scheduler.la lib_profiler.a lib_str.la [...] and then gets this for a segfault: [pid 9810] rename("bat/BBP.dir", "bat/BACKUP/BBP.dir") = 0 [pid 9810] open("/usr/local/MonetDBFast/lib/MonetDB5/mal_init.mal", O_RDONLY) = 15 [pid 9810] open("/usr/local/MonetDBFast/lib/MonetDB5/mal_init.mal", O_RDONLY) = 15 [pid 9810] open("/usr/local [...] [pid 9810] open("/usr/local/MonetDBFast/lib/MonetDB5/bin/lib_blob.so", O_RDONLY) = -1 ENOENT (No such file or directory) [pid 9810] open("/usr/local/MonetDBFast/lib/MonetDB5/blob.mal", O_RDONLY) = 16 [pid 9810] open("/usr/local/MonetDBFast/lib/MonetDB5/blob.mal", O_RDONLY) = 16 [pid 9810] --- SIGSEGV (Segmentation fault) @ 0 (0) --- Process 9810 detached [pid 9811] +++ killed by SIGSEGV +++ [pid 9800] --- SIGCHLD (Child exited) @ 0 (0) --- [pid 9800] wait4(-1, Process 9800 suspended Denham Quantitative Management LP 200 Clarendon Street, 25th Floor Boston, MA 02216 T: 617-587-5812 M: 617-909-7786 omri.schwarz(a)denhamcapital.com NOTICE: This e-mail message is intended for the named recipient(s) above and may contain confidential information. If you are not the intended recipient, you are hereby notified that any dissemination, distribution, or copying of this e-mail and any attachment(s) is strictly prohibited. If you have received this e-mail in error, please immediately notify the sender by replying to this e-mail and delete the message and any attachment(s) from your system. NO REPRESENTATION OR WARRANTY IS MADE TO ANY PERSON AS TO THE ACCURACY OR COMPLETENESS OF THE INFORMATION CONTAINED HEREIN. Denham Capital Management(sm) is a trademark of Denham Capital Management LP.

1 0

Re: [MonetDB-users] PF: "Group By" work-around no longer working?
by Jan Rittinger 14 Jan '09

14 Jan '09

On Jan 14, 2009, at 11:53, jz@uva wrote: > Hi Jan, > > I was also using the GROUP BY construct. > This is my query: ... > let $qid := tijah:queryall-id($query_nexi, $options) > let $nodes := tijah:nodes($qid) > let $result := for $node in $nodes > return > typeswitch ($node) > case $elem-node as element(*) > return > <result> > <num> { (count($node/$elem-node/preceding::*) + 1) + This snippet of XQuery is to complex: 'count($node/$elem-node/...'. Both $node and $elem-node contain the tijah-result nodes (with type node() and element(*), respectively). What you want to write is 'count($elem-node/...'. > > > string-length(tijah:getINEXPath($node)) } </num> > <file> { data($node/$elem-node/ancestor-or-self::EAD/@FILE) } </ > file> > <title> { data($node/$elem-node/ancestor-or-self::EAD/@TITLE) } > </title> ... > > <score> { tijah:score($qid, $node/$elem-node) } </score> > </result> > default return error ("unexpected node") > > let $total := count($result) > return <results total="{$total}"> { > for $res at $rankGlobal in distinct-values($result/file) > let $cs-group := $result[file = $res] > for $cs-group2 at $rank in $cs-group > where exists($cs-group2) this line is unnecessary as there is always exactly one item bound to $cs-group2. > > and $rank >= 1 this line is unnecessary as the positional variable always starts from 1 (and can never be less). > > and $rank <= 8 > and $rankGlobal >= $start > and $rankGlobal <= $end > order by $cs-group2 cast as xs:integer what do you want here? I very much doubt that you are able to atomize an XML snippet, then cast the string-value into a number and still get something meaningful out. > > return > <out id="{$res}">{ $cs-group2 }</out> > } </results> > > With this query, I get different results in MPS and Alg. I guess MPS treats your order by differently---read incorrectly---by ignoring every character data after the first number which accidently is the content of element num. Perhaps your query already works the way you expect it by replacing the ORDER BY by the following snippet: 'order by number($cs-group2/ num)'. Jan -- Jan Rittinger Lehrstuhl Datenbanken und Informationssysteme Wilhelm-Schickard-Institut für Informatik Eberhard-Karls-Universität Tübingen http://www-db.informatik.uni-tuebingen.de/team/rittinger

3 3