- users-list - monetdb.org

[MonetDB-users] Large XML instances with MonetDB/XQuery 4.38.5
by Christian Grün 02 Oct '10

02 Oct '10

Hi everyone, is it still possible to shred large XML instances with MonetDB/XQuery, or do I need more memory for shredding? I was just trying to the 1GB instance of XMark with WinXP 32Bit and 2GB RAM, and this is what I got: MonetDB>shred_doc("1gb.xml", "xmark"); #GDKmmap(536870912) fails, try to free up space [memory in use=36143780,virtual memory in use=1015480320] #GDKmmap(536870912) result [mem=26016940,vm=1005387776] !ERROR: [shred_url]: 1 times inserted nil due to errors at tuples 0@0. !ERROR: [shred_url]: first error was: !ERROR: HEAPextend: failed to extend to 536870912 for 06\632theap !ERROR: shredBAT_append_str: APPEND-STR[_prop_text]( follow murders prov shoes p resent eager delivered defy bend viol here marquis powers complexion pillar cran ts agate ), BUNappend fails !ERROR: CMDshred_url: operation failed. pf:add-doc(...) produces the same output; the XML document itself is well-formed. Thanks, Christian

2 6

[MonetDB-users] Embedded SQL Example Hangs?
by Hering Cheng 23 Sep '10

23 Sep '10

Hi, I am trying to run the example embedded MonetDB server using the June 2010 SP2 release: $ ./Mbeddedsql5 --config=monetdb5.conf --time --dbname=test_db After entering a simple SQL statement on the standard input and pressing <ENTER>, the program hangs at mapi_query. Here is the stack trace: [1] _read(0x5, 0xffffffff7fffdf16, 0x2, 0x0, 0x0, 0x0), at 0xffffffff7a6d3d90 [2] read(0x5, 0xffffffff7fffdf16, 0x2, 0x0, 0x0, 0x0), at 0xffffffff7a6c48b8 [3] pipe_read(s = 0x10015fe50, buf = 0xffffffff7fffdf16, elmsize = 2U, cnt = 1U), line 3194 in "stream.c" [4] stream_readSht(s = 0x10015fe50, val = 0xffffffff7fffdf16), line 2679 in "stream.c" [5] bs_read(ss = 0x10015fcd0, buf = 0x100f7a012, elmsize = 1U, cnt = 8190U), line 2368 in "stream.c" [6] stream_read(s = 0x10015fcd0, buf = 0x100f7a012, elmsize = 1U, cnt = 8190U), line 181 in "stream.c" [7] read_line(mid = 0x100133950), line 2515 in "Mapi.c" [8] read_into_cache(hdl = 0x100f73f30, lookahead = 1), line 2993 in "Mapi.c" [9] mapi_execute_internal(hdl = 0x100f73f30), line 3240 in "Mapi.c" [10] mapi_query(mid = 0x100133950, cmd = 0x100147010 "select count(1) from taq.trade\n"), line 3277 in "Mapi.c" [11] main(argc = 4, av = 0xffffffff7fffe628), line 212 in "prog.c" Here is my MonetDB configuration: $ ./mserver5 --version MonetDB server v5.20.5 (64-bit), based on kernel v1.38.5 (64-bit oids) Release Jun2010-SP2 Copyright (c) 1993-July 2008 CWI Copyright (c) August 2008-2010 MonetDB B.V., all rights reserved Visit http://monetdb.cwi.nl/ for further information Found 64.0GiB available memory, 16 available cpu cores Configured for prefix: /GAAL/chenher/share/monetdb/distro-sparc-jun2010-sp2-64bit Libraries: libpcre: 8.10 2010-06-25 (compiled with 8.10) openssl: OpenSSL 1.0.0a 1 Jun 2010 (compiled with ) libxml2: 2.6.23 (compiled with 2.6.23) Compiled by: chenher@sfouxsrv073 (sparc-sun-solaris2.10) Compilation: cc -m64 -xcode=pic32 -I/GAAL/chenher/share/hans_boehm_gc/distro-sparc-7.1-64bit//include/ -g Linking : /usr/ccs/bin/ld -m64 -L/GAAL/chenher/share/openssl/distro-sparc-1.0.0a-64bit//lib/64/ -L/GAAL/chenher/share/pcre/distro-sparc-8.10-64bit//lib -L/GAAL/chenher/share/hans_boehm_gc/distro-sparc-7.1-64bit//lib/ The database test_db was created by a non-embedded mserver5 using the same configuration file. The non-embedded mserver5 was gracefully shut down just before running the embedded version. Thanks. Hering

1 0

Re: [MonetDB-users] newbies question
by Peter Boncz 21 Sep '10

21 Sep '10

Hello Olivier, MonetDB/XQuery and MonetDB/SQL are currently different servers, based on different MonetDB kernel versions (4 and 5). The new python library you are probably referring to, however, only works for MonetDB5 (thus SQL). MonetDB/XQuery can be accessed using JDBC, XRPC and C (mapi), as per the MonetDB/XQuery API documentation. Peter > -----Original Message----- > From: o.fournier(a)iktios.fr [mailto:o.fournier@iktios.fr] > Sent: Tuesday, September 21, 2010 5:57 PM > To: monetdb-users(a)lists.sourceforge.net > Subject: [MonetDB-users] newbies question > > Hi, > > I find the concept of MonnetDb interresting... > > So, I would like use MonnetDB in SQL and XML mode with python > interface. > Could I use the python MAPI module to request XML mode? > If not, have you plan to do it? > Or what method could I use to access XML Monnetdb from python? > > Thank you > > Olivier Founier > > > > ---------------------------------------------------------------------- > -------- > Start uncovering the many advantages of virtual appliances > and start using them to simplify application deployment and > accelerate your shift to cloud computing. > http://p.sf.net/sfu/novell-sfdev2dev > _______________________________________________ > MonetDB-users mailing list > MonetDB-users(a)lists.sourceforge.net > https://lists.sourceforge.net/lists/listinfo/monetdb-users

1 0

[MonetDB-users] newbies question
by o.fournier＠iktios.fr 21 Sep '10

21 Sep '10

Hi, I find the concept of MonnetDb interresting... So, I would like use MonnetDB in SQL and XML mode with python interface. Could I use the python MAPI module to request XML mode? If not, have you plan to do it? Or what method could I use to access XML Monnetdb from python? Thank you Olivier Founier

1 0

[MonetDB-users] about create index command
by kun ren 17 Sep '10

17 Sep '10

Dear all, I have a problem on create index command, my table is t1,and have two column, c1 and c2. when I execute the sql command "create index index_c1 on t1(c1);" but the MonetDB said "Segmentation fault", what the question? Best regards!

2 2

Re: [MonetDB-users] MonetDB Copy File INTO
by Fabian Groffen 14 Sep '10

14 Sep '10

On 14-09-2010 01:04:41 +0000, mikeschill(a)users.sourceforge.net wrote: > Username: MikeSchill > > I really want to use the MonetDB for our processing but getting > problems trying to do a COPY INTO: Always fails on last line stating > missing '(field delimeter) line (last line) field (last field) > failed to import table > > COPY into STATEDATA from > C:\\Books\\RED.txt' using > delimiters '\t','\n'; > > No idea what I am doing wrong....fails on record 101 if I say COPY 100 > Records... Can you give us any details on what version of MonetDB you are using (such as the release, e.g. Jun2010-SP2)? Also is your data public, or can we get a copy of it to see if we can reproduce the problem? Regards, Fabian Groffen

1 0

[MonetDB-users] 64 bit build with 64 bit OIDs - Invalid data for large string table
by Alex Bo. 07 Sep '10

07 Sep '10

Hi, I built June-SP1 on Windows 64 bit 64 bit OIDs. I loaded a table with one string field with approximately 500M strings. I am running 'Select * from table limit 10', and what I see is this: #1: myVal1 #2: null #3: myVal2 #4: null #5: myVal3 #6: null ... It seems that nulls are inserted between every two consecutive values in the source. The rowcount of the table is correct. I looked inside the dbfarm folders and saw that the theap file is ~26GB, and the tail ~4.6GB. Inserting only 1M of the same data worked fine. Please advise, Thanks. -- View this message in context: http://old.nabble.com/64-bit-build-with-64-bit-OIDs---Invalid-data-for-larg… Sent from the monetdb-users mailing list archive at Nabble.com.

1 0

[MonetDB-users] Building June10-SP1 64 with 64 OID on Windows
by Alex Bo. 06 Sep '10

06 Sep '10

When using MONET_OID64=1 to build June10-SP1 on Windows 64, I get this in the MonetDB5 module: .\..\..\..\..\src\modules\mal\tokenizer.mx(562) : warning C4267: '=' : conversion from 'size_t' to 'int', possible loss of data How can I overcome this? (This did not happen with 32 bit OIDs) Thanks. -- View this message in context: http://old.nabble.com/Building-June10-SP1-64-with-64-OID-on-Windows-tp29632… Sent from the monetdb-users mailing list archive at Nabble.com.

2 2

[MonetDB-users] strPut error with large string tables
by Alex Bo. 02 Sep '10

02 Sep '10

Hi, (June2010 SP1 built on Windows) I loaded a table with a single string column successfully with approximately 500M string values. Running 'Select * from table limit 10' gives me this error: 'strPut: string heaps gets larger than 15GB.' What does this mean? -- View this message in context: http://old.nabble.com/strPut-error-with-large-string-tables-tp29593089p2959… Sent from the monetdb-users mailing list archive at Nabble.com.

3 3

[MonetDB-users] June2010 SP2 failed to build on Windows 64
by Alex Bo. 02 Sep '10

02 Sep '10

Hi, I have successfully built June2010 SP2 on Windows 32 bit. However, the same on 64 bit fails in the Monet module while trying to build 'monet_options.c'. I am getting a bunch of errors related to ws2tcpip.h: C:\Program Files (x86)\Microsoft Visual Studio 8\VC\PlatformSDK\include\ws2tcpip.h(29) : error C2079: 'imr_multiaddr' uses undefined struct 'in_addr' C:\Program Files (x86)\Microsoft Visual Studio 8\VC\PlatformSDK\include\ws2tcpip.h(30) : error C2079: 'imr_interface' uses undefined struct 'in_addr' C:\Program Files (x86)\Microsoft Visual Studio 8\VC\PlatformSDK\include\ws2tcpip.h(38) : error C2079: 'imr_multiaddr' uses undefined struct 'in_addr' C:\Program Files (x86)\Microsoft Visual Studio 8\VC\PlatformSDK\include\ws2tcpip.h(39) : error C2079: 'imr_sourceaddr' uses undefined struct 'in_addr' C:\Program Files (x86)\Microsoft Visual Studio 8\VC\PlatformSDK\include\ws2tcpip.h(40) : error C2079: 'imr_interface' uses undefined struct 'in_addr' C:\Program Files (x86)\Microsoft Visual Studio 8\VC\PlatformSDK\include\ws2tcpip.h(46) : error C2079: 'imsf_multiaddr' uses undefined struct 'in_addr' Please advise. Thanks. -- View this message in context: http://old.nabble.com/June2010-SP2-failed-to-build-on-Windows-64-tp29593272… Sent from the monetdb-users mailing list archive at Nabble.com.

2 3

[MonetDB-users] Where is the right place to fill bug report ?
by Krzysztof Kowalczyk 02 Sep '10

02 Sep '10

Where is the right place to fill a bug report ? I am using sophisticated xquery on MonetDB on Windows x64 but my work has been stopped by a critical bug - probably in Pathfinder. The problem is that Monet fails to compile a query, instead it just take whole available memory and system needs hard restart. Sometimes I can rewrite query to different form with same functionality and this help, but it take a lot of time to find which code part is cause the error. Regards, Krzysztof Kowalczyk

3 4

[MonetDB-users] Joining Across Multiple Databases
by Hering Cheng 31 Aug '10

31 Aug '10

Hi, Is it possible to join two tables residing in different databases in a query? Here, by "database" I mean the result of running "monetdb create <database>". If so, what syntax should we use for the table names? Thanks. Hering Cheng

2 1

[MonetDB-users] Bulk Data Extraction & Embedded mserver5
by Hering Cheng 30 Aug '10

30 Aug '10

Hi, I was wondering if you can give me some ideas on how to retrieve data from MonetDB at very high throughput. The query will be very simple, issued against a single table. This table contains billions of records and I'd like to retrieve the data at a rate of at least one million records per second. From my naive tests with mclient running on the same 16-core machine as mserver5, I am only able to extract the data at about 20,000 records per second with the Feb2010 release. As a baseline case, with the same data in compressed (gzip'd) ASCII form stored in a regular file on the same (SAN) file system as MonetDB, I am able to read at the desired speed of one million records per second. I understand that there is communication overhead between mserver5 and mclient (or whatever client I use). Therefore, one possibility is to embed my application within mserver5. The embedded application basically just needs to be able to issue a SQL (or even MAL) query against the enclosing mserver5 and process the result set. If this is a viable approach, I'd like some guidance on where the hooks are. Thanks. Hering Cheng

5 8

[MonetDB-users] Installation error
by Alexander Rutkowski 30 Aug '10

30 Aug '10

Hi, I tried to install MonetDB on a Debian Linux system with ./monetdb-install.sh --prefix=/export/local-1/public/node/MonetDB --enable-sql --enable-xquery --enable-optimize but I got the error message ... test -z "/export/local-1/public/node/MonetDB/lib" || /bin/mkdir -p "/export/local-1/public/node/MonetDB/lib" /bin/sh ../../libtool --mode=install /usr/bin/install -c libstream.la '/export/local-1/public/node/MonetDB/lib' libtool: install: error: cannot install `libstream.la' to a directory not ending in /usr/local/lib make[6]: *** [install-streamLTLIBRARIES] Error 1 ... installing MonetDB failed I have no root rights so I need the --prefix argument. Thank you very much Alexander

3 3

[MonetDB-users] Announcement: New Jun2010-SP2 bugfix release of MonetDB suite
by Sjoerd Mullender 30 Aug '10

30 Aug '10

The MonetDB team at CWI/MonetDB BV is pleased to announce the Jun2010-SP2 bug fix release of the MonetDB suite of programs. More information (including release notes) on this release is available at <http://monetdb.cwi.nl/Development/Releases/Jun2010/>. The download location is <http://dev.monetdb.org/downloads/>. -- Sjoerd Mullender

1 0

[MonetDB-users] Response time
by Mitja Kleider 14 Aug '10

14 Aug '10

Hi, I am experiencing a huge difference between query response times of "mserver5 --readonly" and plain "mserver5". The difference is in the order of 25 sec vs 25 min. The disks are rather slow and there is a lot of disk write going on. (Not in readonly, obviously.) How can I investigate this further? Regards, Mitja

2 1

[MonetDB-users] Problems with huge SQL queries.
by Manuel Mayr 10 Aug '10

10 Aug '10

Hello everyone, I'm currently using Monet5 for some experiments with ActiveRecord but some SQL queries seem to be too big for the default configuration of monet. The prominent characteristic of the query is a huge IN part in the WHERE clause (nearly 13Kb). $ mclient -d tpch01 -h localhost < q1.sql MAPI = monetdb@localhost:50000 ACTION= read_line QUERY = SELECT states.* FROM states WHERE (states.country_id IN (1,2,3,4,5,6,7,8,9,10,11,12,13,14,15,16,17,18,19,20,21,22,23,24,25,26,27,28,29,30,31,32,33,34,35,36,37,38,39,40,41,42,43,44,45,46,47,48,49,50,51,52,53,54,55,56,57,58,59,60,61,62,63,64,65,66,67,68,69,70,71,72,73,74,75,76,77,78,79,80,81,82,83,84,85,86,87,88,89,90,91,92,93,94,95,96,97,98,99,100,101,102,103,104,105,106,107,108,109,110,111,112,113,114,115,116,117,118,119,120,121,122,123,124,125,126,127,128,129,130,131,132,133,134,135,136,137,138,139,140,141 ERROR = !Connection terminated Is there a parameter that allows me to increase the SQL query buffer? Beste regards, Manuel PS.: I'm not sure if it is really the size of the SQL buffer since the query works when I modify it to run it against sys.tables, but it doesn't seem to return any result. Even prefix the query with "plan" doesn't come back within minutes. Maybe a performance bottleneck in the compiler or planner? -- Manuel Mayr (http://www-db.informatik.uni-tuebingen.de/team/mayr) Lehrstuhl Datenbanksysteme Wilhelm-Schickard Institut für Informatik Eberhard-Karls-Universität Tübingen

7 18

[MonetDB-users] geom performance
by Mitja Kleider 06 Aug '10

06 Aug '10

Hi, I am having trouble with the geom module (Jun2010-SP1). A query constructing a Point(x,y) from longitude (x) and latitude (y) takes much longer than expected, see below: # MonetDB server v5.20.3, based on kernel v1.38.3 # Serving database 'germany3', using 2 threads # Compiled for x86_64-pc-linux-gnu/64bit with 64bit OIDs dynamically linked # MonetDB/SQL module v2.38.3 loaded # MonetDB/GIS module v0.18.3 loaded sql>\d nodes CREATE TABLE "sys"."nodes" ( "id" int NOT NULL DEFAULT next value for "sys"."s_nodes", "long" decimal(10,7), "lat" decimal(10,7), CONSTRAINT "pk_nodes_id" PRIMARY KEY ("id") ); sql>select long,lat from nodes limit 1; +--------------+--------------+ | long | lat | +==============+==============+ | 8.9766191 | 47.7913748 | +--------------+--------------+ 1 tuple (1.0s) sql>select point(8.9766191, 47.7913748); +------------------------------------------------+ | point_single_value | +================================================+ | POINT (8.9766191000000006 47.7913747999999998) | +------------------------------------------------+ 1 tuple (0.897ms) sql>select point(long,lat) from nodes limit 1; +------------------------------------------------+ | point_long | +================================================+ | POINT (8.9766191000000006 47.7913747999999998) | +------------------------------------------------+ 1 tuple (1m 23s) And executing something similar for the whole table caused a segfault after eating a lot of virtual memory: sql>ALTER TABLE nodes ADD geom POINT DEFAULT NULL; sql>UPDATE nodes SET geom = Point(long,lat) WHERE geom IS NULL; In case you would like to try yourself, here is the import script and the parser to get such a database: http://gitorious.org/osm-poi-tools/monetdb/blobs/master/scripts/import.sh http://gitorious.org/osm-poi-tools/monetdb/blobs/master/osmparser/src/osmpa… Thanks for your help, Mitja

2 1

[MonetDB-users] Example of XRPC/Java application
by Ryad Ben-El-Kezadri 05 Aug '10

05 Aug '10

Hello, For whom interested by XRPC and Java GUI, I have posted a video http://www.youtube.com/watch?v=3WzM0ZOJw9k showing the great things that can be done with MonetDB/XQuery Regards Ryad PS: this is a networking application

3 2

[MonetDB-users] How to log every sql query?
by Nicolas Lemay 02 Aug '10

02 Aug '10

Hello, I've been using MonetDB for quite some weeks, but I never find the way to log every sql query my database receive. I've tried googling this, but did not find a lot of docs about that. Is it possible to do this? If yes, how can I accomplish this? Thank you!

1 0

[MonetDB-users] Loading directly into memory.
by Nozhup 31 Jul '10

31 Jul '10

Hi, The first query after a load action takes longer because the new data is still 'cold'. Is there a way in MonetDB to load data directly with a 'hot' status? Thank in advance. -- View this message in context: http://old.nabble.com/Loading-directly-into-memory.-tp29105569p29105569.html Sent from the monetdb-users mailing list archive at Nabble.com.

3 2

Re: [MonetDB-users] MonetDB-users Digest, Vol 50, Issue 11
by Shailendra Mishra 29 Jul '10

29 Jul '10

Hi Niels: I don't care if the log gets written on table creation. I tried the flag but it didn't do anything. The default value is 0. In the FebPS2 version it's a debug flag - actually setting it to 128 doesn't help. On closer examination it appears that it checks for if (debug & 1) then it would write a log. Setting debug to 128, would imply that debug & 1 = 0 and so no debug output is written. Can you tell me where the log writer code is ? Is it gdk_logger.c ? if yes, which is the function that writes the log_write_format, log_write_string, if not which ones. -- Shailendra ------------------------------ > > Message: 5 > Date: Wed, 28 Jul 2010 08:40:06 +0200 > From: Niels Nes <Niels.Nes(a)cwi.nl> > Subject: Re: [MonetDB-users] Creating table with nologging. > To: Communication channel for MonetDB users > <monetdb-users(a)lists.sourceforge.net> > Message-ID: <20100728064006.GA6626(a)cwi.nl> > Content-Type: text/plain; charset=utf-8 > > On Tue, Jul 27, 2010 at 04:28:56PM -0700, Shailendra Mishra wrote: > > I have the need for creating table with nologging enabled. I do > > understand that MonetDB is not meant to be used this way. However, I am > > trying to use it to ingest data off data-feeds. My db lives on a SSD, > > and the only DML that I execute is INSERT. I would appreciate if folks > > familiar with the server code could point me source files ?where I > > could make the change to achieve this. Thanks in advance for your help. > > > > -- Shailendra > > Shailendra > > I haven't tried this in a while but it used to be possible to switch > logging of inserts of using a debug flag (passed to the logger code). > > See the file src/storage/bat/bat_logger.mx > function bl_create. The first argument to logger_create should > be 128 to switch off logging. > > This will only disable that every insert will also write the inserted > values to the log. It still creates a log for your initial create table > statement. > > Indeed this is un documented and far from normal supported usage of > monetdb. > > Niels > >

2 2

[MonetDB-users] MonetDB XQuery newbie
by Stephen Thomas 28 Jul '10

28 Jul '10

I'm a newbie MonetDB user and have two issues out of the box. (I have downloaded and installed the latest stable version, on Mac OS X 10.6.3.) 1. I cannot get the command line history function to work with XQuery. I invoke the server with: Mserver --dbinit="module(pathfinder);" and invoke a client with: mclient -lx --history However, the command line history does not work; hitting the up arrow after a command results in "^[[A" being displayed: shell> mclient -lx --history Welcome to mclient, the MonetDB/XQuery interactive terminal Type \q to quit, \? for a list of available commands xquery>\<test1.xq <hello>World</hello> xquery>^[[A 2. When loading namespaces and modules, and making an error, the error can only be corrected *after* I restart the client and server processes. For example, when I try to import a module with namespace "http://www.foo.edu" but accidentally type "http://www.food.edu", I receive an error message. This is expected. However, if I try again to import the module again, this time with the correct namespace, I get the same exact error message (complete with "http://www.food.edu"). This continues unless I restart both the client and server. Is this a "feature", or am I doing something incorrectly? Thanks for your help, Steve

2 1

[MonetDB-users] Creating table with nologging.
by Shailendra Mishra 28 Jul '10

28 Jul '10

I have the need for creating table with nologging enabled. I do understand that MonetDB is not meant to be used this way. However, I am trying to use it to ingest data off data-feeds. My db lives on a SSD, and the only DML that I execute is INSERT. I would appreciate if folks familiar with the server code could point me source files where I could make the change to achieve this. Thanks in advance for your help. -- Shailendra

2 1

[MonetDB-users] XRPC/execute at not working from mclient
by Ryad Ben-El-Kezadri 27 Jul '10

27 Jul '10

Hello, I found some problems to run a remote Xquery from mclient. The query is the following (it works fine with the Java XRPC API but not with mclient). mclient -lx -s "execute at {'192.168.0.12'} {pf:collections()}" I get the following error. user(win32): password: MAPI = win32@localhost:50000 QUERY = execute at {'192.168.0.12'} {pf:collections()} ERROR = !ERROR: interpret: no matching MIL operator to 'reverse(void)'. !MAYBE YOU MEAN: ! reverse(BAT[any::1,any::2]) : BAT[any::2,any::1] !ERROR: interpret_params: sort(param 1): evaluation error. !ERROR: interpret_params: reverse(param 1): evaluation error. Did anyone had already successfully run such kind of query? PS1: there are 2 Mserver running: one on the local host/one on the remote host. The Java XRPC API does not need to run a Mserver on the local host, but mclient do (else I get a "initiating connection socket failed" error ) PS2:I have Mserver 4.39.0 on the remote linux server and Mserver 4.36.5 on the windows client (The query also fails with client/linux/4.39.0 & server/linux/4.39.0) PS3: the default example provided in the XRPC doc fails (no MIL operator) the same way: import module namespace test = "xrpc-test-function" at " http://192.168.0.12:50001/export/xrpc-mod.xq"; execute at {"192.168.0.12"} {test:add(100, 200)} Regards Ryad

2 1