Constant segmentation faults
Hello. I am evaluating MonetDB to use in our project, but so far have some real problems with MonetDB constantly having segmentation faults. Here is my setup: I am using AWS EC2 instance with 16GB of RAM, monetdb farm is location on ESB volume. Welcome to mclient, the MonetDB/SQL interactive terminal (Jul2015-SP4) Database: MonetDB v11.21.19 (Jul2015-SP4), 'mapi:monetdb://ip-10-0-0-47:50000/mysql-copy' In there I have several tables, among them is one table I am most interested in, problemsview, with total columnsize of 120GB, largest column is around 11GB. The moment I execute “select count(1) from problemsview;” I get segmentation fault: mserver5[1300]: segfault at 0 ip 00007f86a0ca0712 sp 00007f869b6faa80 error 4 in lib_sql.so[7f86a0c72000+199000] Next select * from problemsview limit 10; wait for 20-30 seconds, and then segmentation fault: mserver5[1313]: segfault at 18 ip 00007f68cf2a0726 sp 00007f68ce14ea80 error 4 in lib_sql.so[7f68cf272000+199000] So essentially I cannot do anything, I always get segmentation fault. Can this be fixed somehow? Thank you in advance, Nikita Salnikov-Tarnovski Plumbr Founder and Master Developer @JavaPlumbr/@iNikem
Hi Please provide information on which MonetDB release and platform you are using. Overall, the symptoms provided do not ring a bell on a possible problem. Assuming you were able to create this database, it might by a good idea to trim down the problem by growing the database in a few steps to isolate if size is the problem. You also could check the system logs and/or merovingian.log of your instance to determine if the DBMS received an OOM kill signal from the OS. The last resort is to attach a debugger to the server before you issue the query to catch a stack trace where it happens in the code base. regards, Martin On 06/06/16 18:13, Nikita Salnikov-Tarnovski wrote:
Hello. I am evaluating MonetDB to use in our project, but so far have some real problems with MonetDB constantly having segmentation faults. Here is my setup:
I am using AWS EC2 instance with 16GB of RAM, monetdb farm is location on ESB volume.
Welcome to mclient, the MonetDB/SQL interactive terminal (Jul2015-SP4) Database: MonetDB v11.21.19 (Jul2015-SP4), 'mapi:monetdb://ip-10-0-0-47:50000/mysql-copy'
In there I have several tables, among them is one table I am most interested in, problemsview, with total columnsize of 120GB, largest column is around 11GB.
The moment I execute “select count(1) from problemsview;” I get segmentation fault:
mserver5[1300]: segfault at 0 ip 00007f86a0ca0712 sp 00007f869b6faa80 error 4 in lib_sql.so[7f86a0c72000+199000]
Next select * from problemsview limit 10;
wait for 20-30 seconds, and then segmentation fault: mserver5[1313]: segfault at 18 ip 00007f68cf2a0726 sp 00007f68ce14ea80 error 4 in lib_sql.so[7f68cf272000+199000]
So essentially I cannot do anything, I always get segmentation fault. Can this be fixed somehow?
Thank you in advance,
Nikita Salnikov-Tarnovski Plumbr Founder and Master Developer @JavaPlumbr/@iNikem
_______________________________________________ users-list mailing list users-list@monetdb.org https://www.monetdb.org/mailman/listinfo/users-list
On 06 Jun 2016, at 21:36, Martin Kersten <martin@monetdb.org> wrote:
Hi
Please provide information on which MonetDB release and platform you are using.
Ou, sorry for forgetting this. This is Ubuntu, 14.04.3 LTS. uname -a Linux ip-10-0-0-47 3.13.0-74-generic #118-Ubuntu SMP Thu Dec 17 22:52:10 UTC 2015 x86_64 x86_64 x86_64 GNU/Linux MonetDB version, as I said: 2016-06-06 19:05:27 MSG merovingian[1255]: Merovingian 1.7 (Jul2015-SP4) starting 2016-06-06 19:05:27 MSG merovingian[1255]: monitoring dbfarm /mnt/mysql-copy 2016-06-06 19:05:27 MSG merovingian[1255]: accepting connections on TCP socket 0.0.0.0:50000 2016-06-06 19:05:27 MSG merovingian[1255]: accepting connections on UNIX domain socket /tmp/.s.monetdb.50000 2016-06-06 19:05:27 MSG discovery[1255]: listening for UDP messages on 0.0.0.0:50000 2016-06-06 19:05:27 MSG control[1255]: accepting connections on UNIX domain socket /tmp/.s.merovingian.50000 2016-06-06 19:05:27 MSG discovery[1255]: new neighbour ip-10-0-0-47 (ip-10-0-0-47.eu-west-1.compute.internal) 2016-06-06 19:05:29 MSG discovery[1255]: new database mapi:monetdb://ip-10-0-0-47:50000/mysql-copy (ttl=660s) 2016-06-06 19:05:44 MSG merovingian[1255]: starting database 'mysql-copy', up min/avg/max: 5m/8h/22h, crash average: 1.00 0.90 0.97 (126-4=122) 2016-06-06 19:05:45 MSG mysql-copy[1263]: arguments: /usr/bin/mserver5 --dbpath=/mnt/mysql-copy/mysql-copy --set merovingian_uri=mapi:monetdb://ip-10-0-0-47:50000/mysql-copy --set mapi_open=false --set mapi_port=0 --set mapi_usock=/mnt/mysql-copy/mysql-copy/.mapi.sock --set monet_vault_key=/mnt/mysql-copy/mysql-copy/.vaultkey --set gdk_nr_threads=4 --set max_clients=64 --set sql_optimizer=default_pipe --set monet_daemon=yes 2016-06-06 19:05:45 MSG mysql-copy[1263]: # MonetDB 5 server v11.21.19 "Jul2015-SP4" 2016-06-06 19:05:45 MSG mysql-copy[1263]: # Serving database 'mysql-copy', using 4 threads 2016-06-06 19:05:45 MSG mysql-copy[1263]: # Compiled for x86_64-pc-linux-gnu/64bit with 64bit OIDs and 128bit integers dynamically linked 2016-06-06 19:05:45 MSG mysql-copy[1263]: # Found 15.672 GiB available main-memory.
Overall, the symptoms provided do not ring a bell on a possible problem.
Assuming you were able to create this database, it might by a good idea to trim down the problem by growing the database in a few steps to isolate if size is the problem.
Are there any known limits of tables sizes? Row-wise or size-wise?
You also could check the system logs and/or merovingian.log of your instance to determine if the DBMS received an OOM kill signal from the OS.
merovingan.log: 2016-06-06 19:05:53 MSG merovingian[1255]: database 'mysql-copy' (1263) was killed by signal SIGSEGV dmesg: [ 310.163434] mserver5[1268]: segfault at 0 ip 00007f19deb63712 sp 00007f19ddc12a80 error 4 in lib_sql.so[7f19deb35000+199000] No mentions of oom_killer.
The last resort is to attach a debugger to the server before you issue the query to catch a stack trace where it happens in the code base.
Do you have any how-to doc for that? Thank you for your help, Nikita
regards, Martin
On 06/06/16 18:13, Nikita Salnikov-Tarnovski wrote:
Hello. I am evaluating MonetDB to use in our project, but so far have some real problems with MonetDB constantly having segmentation faults. Here is my setup:
I am using AWS EC2 instance with 16GB of RAM, monetdb farm is location on ESB volume.
Welcome to mclient, the MonetDB/SQL interactive terminal (Jul2015-SP4) Database: MonetDB v11.21.19 (Jul2015-SP4), 'mapi:monetdb://ip-10-0-0-47:50000/mysql-copy'
In there I have several tables, among them is one table I am most interested in, problemsview, with total columnsize of 120GB, largest column is around 11GB.
The moment I execute “select count(1) from problemsview;” I get segmentation fault:
mserver5[1300]: segfault at 0 ip 00007f86a0ca0712 sp 00007f869b6faa80 error 4 in lib_sql.so[7f86a0c72000+199000]
Next select * from problemsview limit 10;
wait for 20-30 seconds, and then segmentation fault: mserver5[1313]: segfault at 18 ip 00007f68cf2a0726 sp 00007f68ce14ea80 error 4 in lib_sql.so[7f68cf272000+199000]
So essentially I cannot do anything, I always get segmentation fault. Can this be fixed somehow?
Thank you in advance,
Nikita Salnikov-Tarnovski Plumbr Founder and Master Developer @JavaPlumbr/@iNikem
_______________________________________________ users-list mailing list users-list@monetdb.org https://www.monetdb.org/mailman/listinfo/users-list
_______________________________________________ users-list mailing list users-list@monetdb.org https://www.monetdb.org/mailman/listinfo/users-list
On 06/06/16 21:10, Nikita Salnikov-Tarnovski wrote:
>> On 06 Jun 2016, at 21:36, Martin Kersten <martin@monetdb.org <mailto:martin@monetdb.org>> wrote:
>>
>> Hi
>>
>> Please provide information on which MonetDB release and platform you are using.
>
> Ou, sorry for forgetting this. This is Ubuntu, 14.04.3 LTS.
>
> uname -a
> Linux ip-10-0-0-47 3.13.0-74-generic #118-Ubuntu SMP Thu Dec 17 22:52:10 UTC 2015 x86_64 x86_64 x86_64 GNU/Linux
>
> MonetDB version, as I said:
>
> 2016-06-06 19:05:27 MSG merovingian[1255]: Merovingian 1.7 (Jul2015-SP4) starting
> 2016-06-06 19:05:27 MSG merovingian[1255]: monitoring dbfarm /mnt/mysql-copy
> 2016-06-06 19:05:27 MSG merovingian[1255]: accepting connections on TCP socket 0.0.0.0:50000
> 2016-06-06 19:05:27 MSG merovingian[1255]: accepting connections on UNIX domain socket /tmp/.s.monetdb.50000
> 2016-06-06 19:05:27 MSG discovery[1255]: listening for UDP messages on 0.0.0.0:50000
> 2016-06-06 19:05:27 MSG control[1255]: accepting connections on UNIX domain socket /tmp/.s.merovingian.50000
> 2016-06-06 19:05:27 MSG discovery[1255]: new neighbour ip-10-0-0-47 (ip-10-0-0-47.eu <http://ip-10-0-0-47.eu>-west-1.compute.internal)
> 2016-06-06 19:05:29 MSG discovery[1255]: new database mapi:monetdb://ip-10-0-0-47:50000/mysql-copy (ttl=660s)
> 2016-06-06 19:05:44 MSG merovingian[1255]: starting database 'mysql-copy', up min/avg/max: 5m/8h/22h, crash average: 1.00 0.90 0.97 (126-4=122)
> 2016-06-06 19:05:45 MSG mysql-copy[1263]: arguments: /usr/bin/mserver5 --dbpath=/mnt/mysql-copy/mysql-copy --set merovingian_uri=mapi:monetdb://ip-10-0-0-47:50000/mysql-copy --set mapi_open=false --set mapi_port=0 --set
> mapi_usock=/mnt/mysql-copy/mysql-copy/.mapi.sock --set monet_vault_key=/mnt/mysql-copy/mysql-copy/.vaultkey --set gdk_nr_threads=4 --set max_clients=64 --set sql_optimizer=default_pipe --set monet_daemon=yes
> 2016-06-06 19:05:45 MSG mysql-copy[1263]: # MonetDB 5 server v11.21.19 "Jul2015-SP4"
> 2016-06-06 19:05:45 MSG mysql-copy[1263]: # Serving database 'mysql-copy', using 4 threads
> 2016-06-06 19:05:45 MSG mysql-copy[1263]: # Compiled for x86_64-pc-linux-gnu/64bit with 64bit OIDs and 128bit integers dynamically linked
> 2016-06-06 19:05:45 MSG mysql-copy[1263]: # Found 15.672 GiB available main-memory.
>
>>
>> Overall, the symptoms provided do not ring a bell on a possible problem.
>>
>> Assuming you were able to create this database, it might by a good idea to trim down
>> the problem by growing the database in a few steps to isolate if size is the problem.
>
> Are there any known limits of tables sizes? Row-wise or size-wise?
There are no limits, except for your diskspace.
>
>> You also could check the system logs and/or merovingian.log of your instance to determine
>> if the DBMS received an OOM kill signal from the OS.
>
> merovingan.log:
> 2016-06-06 19:05:53 MSG merovingian[1255]: database 'mysql-copy' (1263) was killed by signal SIGSEGV
>
> dmesg:
> [ 310.163434] mserver5[1268]: segfault at 0 ip 00007f19deb63712 sp 00007f19ddc12a80 error 4 in lib_sql.so[7f19deb35000+199000]
>
> No mentions of oom_killer.
>
>>
>> The last resort is to attach a debugger to the server before you issue the query to catch
>> a stack trace where it happens in the code base.
>
> Do you have any how-to doc for that?
roughly:
1) start mserver5 using the latest arguments line you can find in the merovingian.log file
2) in a separate window locate the process id of that mserver5 instance
3) attach the debugger:
gdb mserver5 <processId>
continue
4) use mclient to connect to the server and run the query
If it segfaults you can see it in the window 2)
issue a backtrace/bt/where command to find the stack trace.
regards, Martin
>
> Thank you for your help,
>
> Nikita
>
>>
>> regards, Martin
>>
>>
>> On 06/06/16 18:13, Nikita Salnikov-Tarnovski wrote:
>>> Hello. I am evaluating MonetDB to use in our project, but so far have some real problems with MonetDB constantly having segmentation faults. Here is my setup:
>>>
>>> I am using AWS EC2 instance with 16GB of RAM, monetdb farm is location on ESB volume.
>>>
>>> Welcome to mclient, the MonetDB/SQL interactive terminal (Jul2015-SP4)
>>> Database: MonetDB v11.21.19 (Jul2015-SP4), 'mapi:monetdb://ip-10-0-0-47:50000/mysql-copy'
>>>
>>> In there I have several tables, among them is one table I am most interested in, problemsview, with total columnsize of 120GB, largest column is around 11GB.
>>>
>>> The moment I execute “select count(1) from problemsview;” I get segmentation fault:
>>>
>>> mserver5[1300]: segfault at 0 ip 00007f86a0ca0712 sp 00007f869b6faa80 error 4 in lib_sql.so[7f86a0c72000+199000]
>>>
>>> Next
>>> select * from problemsview limit 10;
>>>
>>> wait for 20-30 seconds, and then segmentation fault:
>>> mserver5[1313]: segfault at 18 ip 00007f68cf2a0726 sp 00007f68ce14ea80 error 4 in lib_sql.so[7f68cf272000+199000]
>>>
>>> So essentially I cannot do anything, I always get segmentation fault. Can this be fixed somehow?
>>>
>>> Thank you in advance,
>>>
>>> Nikita Salnikov-Tarnovski
>>> Plumbr Founder and Master Developer
>>> @JavaPlumbr/@iNikem
>>>
>>>
>>>
>>> _______________________________________________
>>> users-list mailing list
>>> users-list@monetdb.org <mailto:users-list@monetdb.org>
>>> https://www.monetdb.org/mailman/listinfo/users-list
>>>
>>
>> _______________________________________________
>> users-list mailing list
>> users-list@monetdb.org <mailto:users-list@monetdb.org>
>> https://www.monetdb.org/mailman/listinfo/users-list
>
>
>
> _______________________________________________
> users-list mailing list
> users-list@monetdb.org
> https://www.monetdb.org/mailman/listinfo/users-list
>
Assuming you were able to create this database, it might by a good idea to trim down the problem by growing the database in a few steps to isolate if size is the problem.
Are there any known limits of tables sizes? Row-wise or size-wise? There are no limits, except for your diskspace.
Then why do you suspect that the problem is connected to the size of the table?
The last resort is to attach a debugger to the server before you issue the query to catch a stack trace where it happens in the code base.
Do you have any how-to doc for that?
roughly: 1) start mserver5 using the latest arguments line you can find in the merovingian.log file 2) in a separate window locate the process id of that mserver5 instance 3) attach the debugger: gdb mserver5 <processId> continue
4) use mclient to connect to the server and run the query If it segfaults you can see it in the window 2) issue a backtrace/bt/where command to find the stack trace.
Tried, but gdb did not tell me anything: (gdb) continue Continuing. Program terminated with signal SIGSEGV, Segmentation fault. The program no longer exists. (gdb) bt No stack. (gdb) backtrace No stack. (gdb) where No stack. (gdb) regards, Nikita
regards, Martin
Thank you for your help,
Nikita
regards, Martin
On 06/06/16 18:13, Nikita Salnikov-Tarnovski wrote:
Hello. I am evaluating MonetDB to use in our project, but so far have some real problems with MonetDB constantly having segmentation faults. Here is my setup:
I am using AWS EC2 instance with 16GB of RAM, monetdb farm is location on ESB volume.
Welcome to mclient, the MonetDB/SQL interactive terminal (Jul2015-SP4) Database: MonetDB v11.21.19 (Jul2015-SP4), 'mapi:monetdb://ip-10-0-0-47:50000/mysql-copy'
In there I have several tables, among them is one table I am most interested in, problemsview, with total columnsize of 120GB, largest column is around 11GB.
The moment I execute “select count(1) from problemsview;” I get segmentation fault:
mserver5[1300]: segfault at 0 ip 00007f86a0ca0712 sp 00007f869b6faa80 error 4 in lib_sql.so[7f86a0c72000+199000]
Next select * from problemsview limit 10;
wait for 20-30 seconds, and then segmentation fault: mserver5[1313]: segfault at 18 ip 00007f68cf2a0726 sp 00007f68ce14ea80 error 4 in lib_sql.so[7f68cf272000+199000]
So essentially I cannot do anything, I always get segmentation fault. Can this be fixed somehow?
Thank you in advance,
Nikita Salnikov-Tarnovski Plumbr Founder and Master Developer @JavaPlumbr/@iNikem
_______________________________________________ users-list mailing list users-list@monetdb.org <mailto:users-list@monetdb.org> <mailto:users-list@monetdb.org <mailto:users-list@monetdb.org>> https://www.monetdb.org/mailman/listinfo/users-list <https://www.monetdb.org/mailman/listinfo/users-list>
_______________________________________________ users-list mailing list users-list@monetdb.org <mailto:users-list@monetdb.org> <mailto:users-list@monetdb.org <mailto:users-list@monetdb.org>> https://www.monetdb.org/mailman/listinfo/users-list <https://www.monetdb.org/mailman/listinfo/users-list>
_______________________________________________ users-list mailing list users-list@monetdb.org <mailto:users-list@monetdb.org> https://www.monetdb.org/mailman/listinfo/users-list <https://www.monetdb.org/mailman/listinfo/users-list>
_______________________________________________ users-list mailing list users-list@monetdb.org <mailto:users-list@monetdb.org> https://www.monetdb.org/mailman/listinfo/users-list <https://www.monetdb.org/mailman/listinfo/users-list>
Debugging requires a debug build of MonetDB. I suppose you installed the binary distribution (.deb package / via apt-get)? We do not provide any debug builds as distributions. You'd need to compile MonetDB from sources yourself, configured with --enable-debug; cf., https://www.monetdb.org/Developers/SourceCompile or https://dev.monetdb.org/hg/MonetDB/file/tip/HowToStart.rst Best, Stefan ----- On Jun 7, 2016, at 6:53 AM, Nikita Salnikov-Tarnovski nikem@plumbr.eu wrote:
Assuming you were able to create this database, it might by a good idea to trim down the problem by growing the database in a few steps to isolate if size is the problem.
Are there any known limits of tables sizes? Row-wise or size-wise? There are no limits, except for your diskspace.
Then why do you suspect that the problem is connected to the size of the table?
The last resort is to attach a debugger to the server before you issue the query to catch a stack trace where it happens in the code base.
Do you have any how-to doc for that? roughly: 1) start mserver5 using the latest arguments line you can find in the merovingian.log file 2) in a separate window locate the process id of that mserver5 instance 3) attach the debugger: gdb mserver5 <processId> continue
4) use mclient to connect to the server and run the query If it segfaults you can see it in the window 2) issue a backtrace/bt/where command to find the stack trace.
Tried, but gdb did not tell me anything:
(gdb) continue Continuing.
Program terminated with signal SIGSEGV, Segmentation fault. The program no longer exists. (gdb) bt No stack. (gdb) backtrace No stack. (gdb) where No stack. (gdb)
regards, Nikita
regards, Martin
Thank you for your help,
Nikita
regards, Martin
On 06/06/16 18:13, Nikita Salnikov-Tarnovski wrote:
Hello. I am evaluating MonetDB to use in our project, but so far have some real problems with MonetDB constantly having segmentation faults. Here is my setup:
I am using AWS EC2 instance with 16GB of RAM, monetdb farm is location on ESB volume.
Welcome to mclient, the MonetDB/SQL interactive terminal (Jul2015-SP4) Database: MonetDB v11.21.19 (Jul2015-SP4), 'mapi:monetdb://ip-10-0-0-47:50000/mysql-copy'
In there I have several tables, among them is one table I am most interested in, problemsview, with total columnsize of 120GB, largest column is around 11GB.
The moment I execute “select count(1) from problemsview;” I get segmentation fault:
mserver5[1300]: segfault at 0 ip 00007f86a0ca0712 sp 00007f869b6faa80 error 4 in lib_sql.so[7f86a0c72000+199000]
Next select * from problemsview limit 10;
wait for 20-30 seconds, and then segmentation fault: mserver5[1313]: segfault at 18 ip 00007f68cf2a0726 sp 00007f68ce14ea80 error 4 in lib_sql.so[7f68cf272000+199000]
So essentially I cannot do anything, I always get segmentation fault. Can this be fixed somehow?
Thank you in advance,
Nikita Salnikov-Tarnovski Plumbr Founder and Master Developer @JavaPlumbr/@iNikem
_______________________________________________ users-list mailing list users-list@monetdb.org < mailto:users-list@monetdb.org > https://www.monetdb.org/mailman/listinfo/users-list
_______________________________________________ users-list mailing list users-list@monetdb.org < mailto:users-list@monetdb.org > https://www.monetdb.org/mailman/listinfo/users-list
_______________________________________________ users-list mailing list users-list@monetdb.org https://www.monetdb.org/mailman/listinfo/users-list
_______________________________________________ users-list mailing list users-list@monetdb.org https://www.monetdb.org/mailman/listinfo/users-list
_______________________________________________ users-list mailing list users-list@monetdb.org https://www.monetdb.org/mailman/listinfo/users-list
-- | Stefan.Manegold@CWI.nl | DB Architectures (DA) | | www.CWI.nl/~manegold/ | Science Park 123 (L321) | | +31 (0)20 592-4212 | 1098 XG Amsterdam (NL) |
participants (3)
-
Martin Kersten -
Nikita Salnikov-Tarnovski -
Stefan Manegold