- users-list - monetdb.org

[MonetDB-users] Concurrent user threads
by Vashti_Ragoonath＠ibi.com 10 Dec '09

10 Dec '09

Hi, I read in the SQL Reference Manual that "The number of concurrent user threads is a configuration parameter.", but no information on the name or location or how to change it. Nor is there any info on what the default might be. I tried running multiple users and noticed that the performance degradation increased dramatically as the load increased. I looked at the Server and SQL manuals without luck. Thanks, Tah

3 3

[MonetDB-users] file system usage
by Matthew Jones 10 Dec '09

10 Dec '09

Hey folks, I'm wondering if you can tell me how MonetDB manages its on-disk database. During normal operation I was seeing the database hovering at around 55G, after I stopped it to take a backup it was at 70G, but immediately after the restart it dropped down to 35G. 73, Matthew W. Jones (KI4ZIB) http://matburt.net

1 0

[MonetDB-users] lib_replication
by Vashti_Ragoonath＠ibi.com 10 Dec '09

10 Dec '09

Hi, I was able to load up and run the MonetDB5-SQL-Installer-x86_64-20091023.msi. However, when I tried the MonetDB5-SQL-Installer-x86_64-20091119.msi I received an error about the server failing to start because it could not find lib_replication.dll. There is no indication in the Readme.html that this has to be built and it's not in the 1119 package. I tried using the 1119 because I created some compressed files (gz files created with gzip) that I wanted to load using COPY INTO and read that this is supported on Windows in the 1119 version. In the 1123 version it generated an error when loading the GZ files about not being able to locate the files - a bogus error it seems since the file path was correct. I looked on Google, forums etc but no other reports. I reloaded the 1123 to make sure that was still working and it is. I hope I did not miss a basic step or something in which case I apologize profusely! Thanks, Tah

2 1

Re: [MonetDB-users] monetsql python and blobs
by Gijs Molenaar 10 Dec '09

10 Dec '09

On 12/09/2009 11:49 AM, Lazaros Koromilas wrote: > just a problem i encountered. > see attachment. > Thanks for the patch. I will test and include it. One question, why is the monetdb python API is such a strange location in your installation? Did you manually move it to MonetSQLdb? - gijs

1 0

[MonetDB-users] monetsql python and blobs
by Lazaros Koromilas 09 Dec '09

09 Dec '09

just a problem i encountered. see attachment. thanx, lazaros. -- http://www.ics.forth.gr/~koromil

1 0

[MonetDB-users] enabling history
by Nafees Ur Rehman 02 Dec '09

02 Dec '09

Hi, I am enabling history (set history=true;) but it gives me an exception that two procedures are not available. Can someone guide me on that? Regards Nafees Ur Rehman University of Konstanz

3 2

[MonetDB-users] Announcement: New Nov2009 feature release of MonetDB suite
by Stefan Manegold 29 Nov '09

29 Nov '09

Dear all, The MonetDB team at CWI/MonetDB BV are pleased to announce the new "Nov2009" feature release of the MonetDB suite of programs. This feature release is backward compatible with the previous releases (Nov2008 through Aug2009-SP2). Caveat 1: In MonetDB/XQuery, we extended and improved the use of value indices on comparisions between a value and an XPath ending in text() or attribute access (e.g., doc(x)//x/y[z/text() = "foo"] and for $x in doc//x/y where $x/@z = "bar" return $x); such queries do not need scanning data, but start at the index and navigate backwards from there. This required a change in the physical database storage. While the Nov2009 release of MonetDB/XQuery can still read older (pre-Nov2009) databases, it is recommended to (back-up and) re-shred all documents to benefit from the new indices and achieve optimal performance. Caveat 2: When upgrading from a release older than May2009 (i.e., Nov2008 through Feb2009-SP2), you need to move your write-ahead log following the "recommended" instructions for SQL (<http://monetdb.cwi.nl/Development/Releases/May2009/#SQL>) and/or XQuery (<http://monetdb.cwi.nl/Development/Releases/May2009/#XQuery>). More information on the new release is available at <http://monetdb.cwi.nl/Development/Releases/Nov2009/>. The sources and binaries are available at <http://monetdb.cwi.nl/Download/>. Enjoy! The MonetDB team at CWI/MonetDB BV. -- | Dr. Stefan Manegold | mailto:Stefan.Manegold@cwi.nl | | CWI, P.O.Box 94079 | http://www.cwi.nl/~manegold/ | | 1090 GB Amsterdam | Tel.: +31 (20) 592-4212 | | The Netherlands | Fax : +31 (20) 592-4312 |

1 0

[MonetDB-users] SABAOTHgetMyStatus Too many open files
by Matthew Jones 28 Nov '09

28 Nov '09

Hey Folks, we are continuing to get deadlocks about once a day and I've noticed this in the logfile: 2009-11-28 14:31:05 ERR reporting_primary[31485]: !SABAOTHgetMyStatus: IOException:sabaoth.getStatus:Unable to open directory /opt/MonetDB5/dbfarm: Too many open files Not sure how many open files it had when it went south, but immediately after a restart I saw this in files-nr: # cat /proc/sys/fs/file-nr 1408 0 3164775 That third number being the 'files-max' on the system currently... I think that must be being dynamically set. This is with the latest release August2009-SP2, the net effect is a deadlock on all connections to merovingian and a segfault of the mserver process. Neils, this is what led up to that 'log' file being truncated in sql_logs a few days ago... this message occurred just before this happened. I've restarted my database and should see the problem reproduce itself again within 24 hours. I am recording the number of file descriptors in use every minute by mserver, merovingian, and overall for the system so I can know what the numbers were leading up to it. Is this a file descriptor leak in Monet or is my files-max set too low (even though it appears to be dynamic)? 73, Matthew W. Jones (KI4ZIB) http://matburt.net

1 0

[MonetDB-users] COPY INTO memory issues
by Andreas Streichardt 26 Nov '09

26 Nov '09

Hi, i had a look at monetdb a while (a year or so) ago and stumbled upon a few crashes/memory issues back then. Apart from that it made a very promising impression to me. Today i thought i could give it a try again to see if these issues have been fixed and found out that copy into still seems to have memory issues and i wonder if i am the only one who hits such issues. I am experimeting with dummy data created by a script. Data is created the following way: 20 smallints: Values 1-10. 100 tinyints: Values 0-1 20 ints: Values 1-10000 40 varchars: md5 of some random value 20 floats: rand(1000000)/1000000 I created a CSV file with ~2.1 million rows and then tried to insert the data. My first attempt was: sql>copy into md_data FROM '/home/mop/performance/hans.csv' delimiters ',','\n' null as ''; MAPI = monetdb@localhost:50000 ACTION= read_line QUERY = copy into md_data FROM '/home/mop/performance/hans.csv' delimiters ',','\n' null as ''; ERROR = Connection terminated This was of course the quick and dirty way. What happened in the background was that monetdb consumed all memory. My system is a debian lenny 64bit. Server has 4GB and decent CPUs. Why does it need all my memory here? I then did the same but loaded only parts of the file (1000000 rows). sql>copy 1000000 records into md_data FROM '/home/mop/performance/hans.csv' delimiters ',','\n' null as '' ; This worked but the server used ~2-3GB of my RAM during the process and what's even more important it used the same amount even though the copy process finished. Why does it use the RAM even though it is not used anymore? Then i tried to insert 1000000 rows more and my expectation was that monetdb would reuse the memory it allocated (and didn't free) in the first loading process. This wasn't the fact. Again monetdb consumed all my memory and when loading finished it indicated that it was still doing something (CPU @ ~20%). When i tried a: sql>SELECT COUNT(*) FROM md_data; it hang completely. Same when trying to establish a new connection. I stopped the server and restarted again. Everything was fine again. I inserted the rest of the data (memory raised again but there were only 100000 rows left so there wasn't any problem this time). Am i the only one hitting such issues? One could argue that i should import less records at once but even then. There seems to be some nasty memleak in the background. Or is this a configuration issue? Or am i doing something completely wrong? I am using the .debs: http://monetdb.cwi.nl/downloads/Debian/ Thanks in advance, Andreas Streichardt

2 3

Re: [MonetDB-users] COPY INTO memory issues
by Jens Oehlschlägel 25 Nov '09

25 Nov '09

> > This wasn't the fact. Again monetdb consumed all my memory and when loading > > finished it indicated that it was still doing something (CPU @ ~20%). > log records have to be written, your OS may kick in to flush dirty pages, > reloading swapped out processes, .... The following might not apply to Linux, but still: under windows, suddenly flushing dirty pages from large memory mapped files can severly degrade the system performance ("stalled"). I know windows has settings to limit the amount of dirty pages that may accumulate without flushing. Maybe Linux has something similar and maybe this helps here. Please keep me in the loop if you try this route. Jens

1 0