On Jan 22, 2008 8:48 AM, Bob Ippolito <bob@redivi.com> wrote:
So here's the core dump for our ETL test with --nightly=stable. We have not yet tried to restart mserver5. Here's some stats on the data dirs:
$ du -sh * 10G bat 4.0K box 5.6G core.6950
$ find bat|wc -l 777933
We've run into another error during ETL processing. I've posted details similar to those above. $ du -sh * 4.2G bat 4.0K box $ find bat|wc -l 3553 The screen output from the mserver5 process is below, thanks for any help you can provide: Travis !WARNING: GDKlockHome: created directory /data/db/monetdb/mochiads_stats/ !WARNING: GDKlockHome: ignoring empty or invalid .gdk_lock. !WARNING: BBPdir: initializing BBP. # MonetDB server v5.2.3, based on kernel v1.20.1 # Serving database 'mochiads_stats' # Compiled for x86_64-unknown-linux-gnu/64bit with 64bit OIDs dynamically linked # Copyright (c) 1993-2007 CWI, all rights reserved # Visit http://monetdb.cwi.nl/ for further information #warning: please don't forget to set your vault key! #(see /mochi/etc/monetdb5.conf) # Listening for connection requests on mapi:monetdb://127.0.0.1:50000/ # MonetDB/SQL module v2.20.3 loaded
#MALexit: clients still active #MALexit: clients still active #MALexit: server forced exit 0 finishing 0 claimed 1 waiting # MonetDB server v5.2.3, based on kernel v1.20.1 # Serving database 'mochiads_stats' # Compiled for x86_64-unknown-linux-gnu/64bit with 64bit OIDs dynamically
# MonetDB server v5.2.3, based on kernel v1.20.1 # Serving database 'mochiads_stats' # Compiled for x86_64-unknown-linux-gnu/64bit with 64bit OIDs dynamically
[K>\q # MonetDB server v5.2.3, based on kernel v1.20.1 # Serving database 'mochiads_stats' # Compiled for x86_64-unknown-linux-gnu/64bit with 64bit OIDs dynamically linked # Copyright (c) 1993-2007 CWI, all rights reserved # Visit http://monetdb.cwi.nl/ for further information #warning: please don't forget to set your vault key! #(see /mochi/etc/monetdb5.conf) # Listening for connection requests on mapi:monetdb://127.0.0.1:50000/ # MonetDB/SQL module v2.20.3 loaded linked # Copyright (c) 1993-2007 CWI, all rights reserved # Visit http://monetdb.cwi.nl/ for further information #warning: please don't forget to set your vault key! #(see /mochi/etc/monetdb5.conf) # Listening for connection requests on mapi:monetdb://127.0.0.1:50000/ # MonetDB/SQL module v2.20.3 loaded linked # Copyright (c) 1993-2007 CWI, all rights reserved # Visit http://monetdb.cwi.nl/ for further information #warning: please don't forget to set your vault key! #(see /mochi/etc/monetdb5.conf) # Listening for connection requests on mapi:monetdb://127.0.0.1:50000/ # MonetDB/SQL module v2.20.3 loaded
*** glibc detected *** //mochi/bin/mserver5: free(): invalid pointer: 0x00002aaaf9969008 *** ======= Backtrace: ========= /lib/libc.so.6[0x2aaaff2f8b23] /lib/libc.so.6(cfree+0x8c)[0x2aaaff2fc26c] /mochi/lib/libbat.so.0(GDKfree+0x2e)[0x2aaafd8d163e] /mochi/lib/libbat.so.0(HEAPfree+0xad)[0x2aaafd86dd2d] /mochi/lib/libbat.so.0(BATdelete+0x24b)[0x2aaafd9134ab] /mochi/lib/libbat.so.0[0x2aaafd86a52f] /mochi/lib/libbat.so.0[0x2aaafd86b5f4] /mochi/lib/libmal.so.0[0x2aaafcfc2ae0] /mochi/lib/libmal.so.0(callMAL+0x25b)[0x2aaafcfc41cb] /mochi/lib/MonetDB5/lib/lib_sql.so(SQLexecutePrepared+0x25c)[0x2aaab266b19c] /mochi/lib/MonetDB5/lib/lib_sql.so(SQLengineIntern+0x21d)[0x2aaab266cd0d] /mochi/lib/libmal.so.0(runScenarioBody+0x14d)[0x2aaafcfe305d] /mochi/lib/libmal.so.0(runScenario+0x3a)[0x2aaafcfe313a] /mochi/lib/libmal.so.0(MSserveClient+0x58)[0x2aaafcfbe1d8] /lib/libpthread.so.0[0x2aaafe6782a5] /lib/libc.so.6(clone+0x6d)[0x2aaaff35861d] ======= Memory map: ======== 00400000-00404000 r-xp 00000000 fe:00 2115695 /mochi/bin/mserver5 00604000-00605000 rw-p 00004000 fe:00 2115695 /mochi/bin/mserver5 00605000-3fff6000 rw-p 00605000 00:00 0 [heap] 40000000-40001000 ---p 40000000 00:00 0 40001000-40801000 rw-p 40001000 00:00 0 40801000-40802000 ---p 40801000 00:00 0 40802000-41002000 rw-p 40802000 00:00 0 41002000-41003000 ---p 41002000 00:00 0 41003000-41803000 rw-p 41003000 00:00 0 41803000-41804000 ---p 41803000 00:00 0 41804000-42004000 rw-p 41804000 00:00 0 42004000-42005000 ---p 42004000 00:00 0 42005000-42805000 rw-p 42005000 00:00 0 2aaaaaaab000-2aaaaaaad000 rw-p 2aaaaaaab000 00:00 0 2aaaaabad000-2aaaaabb1000 r-xp 00000000 fe:00 2085181 /mochi/lib/MonetDB5/lib/lib_mdb.so.0.0.0 2aaaaabb1000-2aaaaadb1000 ---p 00004000 fe:00 2085181 /mochi/lib/MonetDB5/lib/lib_mdb.so.0.0.0 2aaaaadb1000-2aaaaadb2000 rw-p 00004000 fe:00 2085181 /mochi/lib/MonetDB5/lib/lib_mdb.so.0.0.0 2aaaaadb2000-2aaaaadb4000 r-xp 00000000 fe:00 2084480 /mochi/lib/MonetDB5/lib/lib_xml.so.0.0.0 2aaaaadb4000-2aaaaafb3000 ---p 00002000 fe:00 2084480 /mochi/lib/MonetDB5/lib/lib_xml.so.0.0.0 2aaaaafb3000-2aaaaafb4000 rw-p 00001000 fe:00 2084480 /mochi/lib/MonetDB5/lib/lib_xml.so.0.0.0 2aaaaafbb000-2aaaab03c000 r-xp 00000000 fe:00 540697 /lib/libm-2.5.so 2aaaab03c000-2aaaab23b000 ---p 00081000 fe:00 540697 /lib/libm-2.5.so 2aaaab23b000-2aaaab23d000 rw-p 00080000 fe:00 540697 /lib/libm-2.5.so 2aaaab23d000-2aaaab24b000 r-xp 00000000 fe:00 2016109 /mochi/lib/libMapi.so.0.0.0 2aaaab24b000-2aaaab44a000 ---p 0000e000 fe:00 2016109 /mochi/lib/libMapi.so.0.0.0 2aaaab44a000-2aaaab44b000 rw-p 0000d000 fe:00 2016109 /mochi/lib/libMapi.so.0.0.0 2aaaab44b000-2aaaab44c000 r-xp 00000000 fe:00 2085120 /mochi/lib/MonetDB5/lib/lib_unix.so.0.0.0 2aaaab44c000-2aaaab64b000 ---p 00001000 fe:00 2085120 /mochi/lib/MonetDB5/lib/lib_unix.so.0.0.0 2aaaab64b000-2aaaab64c000 rw-p 00000000 fe:00 2085120 /mochi/lib/MonetDB5/lib/lib_unix.so.0.0.0 2aaaab64c000-2aaaab64e000 r-xp 00000000 fe:00 2085099 /mochi/lib/MonetDB5/lib/lib_lock.so.0.0.0 2aaaab64e000-2aaaab84d000 ---p 00002000 fe:00 2085099 /mochi/lib/MonetDB5/lib/lib_lock.so.0.0.0 2aaaab84d000-2aaaab84e000 rw-p 00001000 fe:00 2085099 /mochi/lib/MonetDB5/lib/lib_lock.so.0.0.0 2aaaab84e000-2aaaab84f000 r-xp 00000000 fe:00 2085118 /mochi/lib/MonetDB5/lib/lib_sema.so.0.0.0 2aaaab84f000-2aaaaba4f000 ---p 00001000 fe:00 2085118 /mochi/lib/MonetDB5/lib/lib_sema.so.0.0.0 2aaaaba4f000-2aaaaba50000 rw-p 00001000 fe:00 2085118 /mochi/lib/MonetDB5/lib/lib_sema.so.0.0.0 2aaaaba50000-2aaaaba55000 r-xp 00000000 fe:00 2085116 /mochi/lib/MonetDB5/lib/lib_mmath.so.0.0.0 2aaaaba55000-2aaaabc54000 ---p 00005000 fe:00 2085116 /mochi/lib/MonetDB5/lib/lib_mmath.so.0.0.0 2aaaabc54000-2aaaabc55000 rw-p 00004000 fe:00 2085116 /mochi/lib/MonetDB5/lib/lib_mmath.so.0.0.0 2aaaabc55000-2aaaabc58000 r-xp 00000000 fe:00 2084470 /mochi/lib/MonetDB5/lib/lib_streams.so.0.0.0 2aaaabc58000-2aaaabe58000 ---p 00003000 fe:00 2084470 /mochi/lib/MonetDB5/lib/lib_streams.so.0.0.0 2aaaabe58000-2aaaabe59000 rw-p 00003000 fe:00 2084470 /mochi/lib/MonetDB5/lib/lib_streams.so.0.0.0 2aaaabe59000-2aaaabe69000 r-xp 00000000 fe:00 2085061 /mochi/lib/MonetDB5/lib/lib_bat5.so.0.0.0 2aaaabe69000-2aaaac069000 ---p 00010000 fe:00 2085061 /mochi/lib/MonetDB5/lib/lib_bat5.so.0.0.0 2aaaac069000-2aaaac06a000 rw-p 00010000 fe:00 2085061 /mochi/lib/MonetDB5/lib/lib_bat5.so.0.0.0 2aaaac06a000-2aaaac06d000 r-xp 00000000 fe:00 2085124 /mochi/lib/MonetDB5/lib/lib_batExtensions.so.0.0.0 2aaaac06d000-2aaaac26c000 ---p 00003000 fe:00 2085124 /mochi/lib/MonetDB5/lib/lib_batExtensions.so.0.0.0 2aaaac26c000-2aaaac26d000 rw-p 00002000 fe:00 2085124 /mochi/lib/MonetDB5/lib/lib_batExtensions.so.0.0.0 2aaaac26d000-2aaaac270000 r-xp 00000000 fe:00 2085202 /mochi/lib/MonetDB5/lib/lib_pcre.so.0.0.0 2aaaac270000-2aaaac46f000 ---p 00003000 fe:00 2085202 /mochi/lib/MonetDB5/lib/lib_pcre.so.0.0.0 2aaaac46f000-2aaaac470000 rw-p 00002000 fe:00 2085202 /mochi/lib/MonetDB5/lib/lib_pcre.so.0.0.0 2aaaac470000-2aaaac49e000 r-xp 00000000 fe:00 2084507 /mochi/lib/MonetDB5/lib/lib_algebra.so.0.0.0 2aaaac49e000-2aaaac69d000 ---p 0002e000 fe:00 2084507 /mochi/lib/MonetDB5/lib/lib_algebra.so.0.0.0 2aaaac69d000-2aaaac69e000 rw-p 0002d000 fe:00 2084507 /mochi/lib/MonetDB5/lib/lib_algebra.so.0.0.0 2aaaac69e000-2aaaac69f000 r-xp 00000000 fe:00 2085122 /mochi/lib/MonetDB5/lib/lib_algebraExtensions.so.0.0.0 2aaaac69f000-2aaaac89f000 ---p 00001000 fe:00 2085122 /mochi/lib/MonetDB5/lib/lib_algebraExtensions.so.0.0.0 2aaaac89f000-2aaaac8a0000 rw-p 00001000 fe:00 2085122 /mochi/lib/MonetDB5/lib/lib_algebraExtensions.so.0.0.0 2aaaac8a0000-2aaaac8c3000 r-xp 00000000 fe:00 2085097 /mochi/lib/MonetDB5/lib/lib_calc.so.0.0.0 2aaaac8c3000-2aaaacac2000 ---p 00023000 fe:00 2085097 /mochi/lib/MonetDB5/lib/lib_calc.so.0.0.0 2aaaacac2000-2aaaacac3000 rw-p 00022000 fe:00 2085097 /mochi/lib/MonetDB5/lib/lib_calc.so.0.0.0 2aaaacac3000-2aaaacac7000 r-xp 00000000 fe:00 2085119 /mochi/lib/MonetDB5/lib/lib_status.so.0.0.0 2aaaacac7000-2aaaaccc7000 ---p 00004000 fe:00 2085119 /mochi/lib/MonetDB5/lib/lib_status.so.0.0.0 2aaaaccc7000-2aaaaccc8000 rw-p 00004000 fe:00 2085119 /mochi/lib/MonetDB5/lib/lib_status.so.0.0.0 2aaaaccc8000-2aaaacd00000 r-xp 00000000 fe:00 2085098 /mochi/lib/MonetDB5/lib/lib_group.so.0.0.0 2aaaacd00000-2aaaacf00000 ---p 00038000 fe:00 2085098 /mochi/lib/MonetDB5/lib/lib_group.so.0.0.0 2aaaacf00000-2aaaacf01000 rw-p 00038000 fe:00 2085098 /mochi/lib/MonetDB5/lib/lib_group.so.0.0.0 2aaaacf01000-2aaaad049000 r-xp 00000000 fe:00 2084493 /mochi/lib/MonetDB5/lib/lib_aggr.so.0.0.0 2aaaad049000-2aaaad248000 ---p 00148000 fe:00 2084493 /mochi/lib/MonetDB5/lib/lib_aggr.so.0.0.0 2aaaad248000-2aaaad249000 rw-p 00147000 fe:00 2084493 /mochi/lib/MonetDB5/lib/lib_aggr.so.0.0.0 2aaaad249000-2aaaad24f000 r-xp 00000000 fe:00 2084562 /mochi/lib/MonetDB5/lib/lib_array.so.0.0.0 2aaaad24f000-2aaaad44e000 ---p 00006000 fe:00 2084562 /mochi/lib/MonetDB5/lib/lib_arra