- users-list - monetdb.org

BULK UPLOAD very poor performance
by Ariel Abadi 24 Mar '20

24 Mar '20

Hi! Maybe somebody can help me, please! I'm trying to upload into Monet a file with 14MM records. Is taking more than *24 minutes.* It is a fresh new database. Enclosed you will find the server configuration. Monetdb Version Nov2019-SP3 *[root@mn tmp]$ lscpu* Architecture: x86_64 CPU op-mode(s): 32-bit, 64-bit Byte Order: Little Endian CPU(s): 40 On-line CPU(s) list: 0-39 Thread(s) per core: 2 Core(s) per socket: 10 Socket(s): 2 NUMA node(s): 2 Vendor ID: GenuineIntel CPU family: 6 Model: 85 Model name: Intel(R) Xeon(R) Silver 4114 CPU @ 2.20GHz Stepping: 4 CPU MHz: 800.479 BogoMIPS: 4400.00 Virtualization: VT-x L1d cache: 32K L1i cache: 32K L2 cache: 1024K L3 cache: 14080K NUMA node0 CPU(s): 0,2,4,6,8,10,12,14,16,18,20,22,24,26,28,30,32,34,36,38 NUMA node1 CPU(s): 1,3,5,7,9,11,13,15,17,19,21,23,25,27,29,31,33,35,37,39 Flags: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx pdpe1gb rdtscp lm constant_tsc art arch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc cpuid aperfmperf pni pclmulqdq dtes64 monitor ds_cpl vmx smx est tm2 ssse3 sdbg fma cx16 xtpr pdcm pcid dca sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer aes xsave avx f16c rdrand lahf_lm abm 3dnowprefetch cpuid_fault epb cat_l3 cdp_l3 invpcid_single pti intel_ppin ssbd mba ibrs ibpb stibp tpr_shadow vnmi flexpriority ept vpid ept_ad fsgsbase tsc_adjust bmi1 hle avx2 smep bmi2 erms invpcid rtm cqm mpx rdt_a avx512f avx512dq rdseed adx smap clflushopt clwb intel_pt avx512cd avx512bw avx512vl xsaveopt xsavec xgetbv1 xsaves cqm_llc cqm_occup_llc cqm_mbm_total cqm_mbm_local dtherm ida arat pln pts pku ospke md_clear flush_l1d *[root@mn tmp]$ cat /proc/meminfo* MemTotal: 83886080 kB MemFree: 10535812 kB MemAvailable: 80159744 kB Buffers: 0 kB Cached: 69623932 kB SwapCached: 0 kB Active: 6625436 kB Inactive: 63737872 kB Active(anon): 324456 kB Inactive(anon): 417184 kB Active(file): 6300980 kB Inactive(file): 63320688 kB Unevictable: 0 kB Mlocked: 5320 kB SwapTotal: 8388604 kB SwapFree: 8381988 kB Dirty: 220 kB Writeback: 0 kB AnonPages: 13626052 kB Mapped: 186368 kB Shmem: 1800 kB KReclaimable: 4134960 kB Slab: 0 kB SReclaimable: 0 kB SUnreclaim: 0 kB KernelStack: 16720 kB PageTables: 56500 kB NFS_Unstable: 0 kB Bounce: 0 kB WritebackTmp: 0 kB CommitLimit: 74123472 kB Committed_AS: 18337812 kB VmallocTotal: 34359738367 kB VmallocUsed: 415836 kB VmallocChunk: 0 kB Percpu: 39552 kB HardwareCorrupted: 0 kB AnonHugePages: 0 kB ShmemHugePages: 0 kB ShmemPmdMapped: 0 kB CmaTotal: 0 kB CmaFree: 0 kB HugePages_Total: 0 HugePages_Free: 0 HugePages_Rsvd: 0 HugePages_Surp: 0 Hugepagesize: 2048 kB Hugetlb: 0 kB DirectMap4k: 816396 kB DirectMap2M: 82513920 kB DirectMap1G: 52428800 kB

2 2

Slowdown behaviour after loading large amounts of data
by stephenb 23 Mar '20

23 Mar '20

We believe we might be observing the behaviour described here: https://www.monetdb.org/blog/monetdb-sql-transaction-management-scheme Specifically “At certain intervals, if the database is idle, the store manager will merge the updates from the WAL into the primary storage” We perform some heavy updates to the database (we have experimented with putting these in a single transaction or having separate transactions for each, but the behaviour is the same). The updates are basically dropping tables, recreating them and then a COPY INTO. Once the final load has finished, and has been committed, for around 30 seconds we can query the database, but then for a period of several minutes it basically locks up. I wonder if there is any way to control this behaviour? Regards Steve

1 0

stethoscope tool
by Suresh Devarakonda 12 Mar '20

12 Mar '20

Thank you Sjoerd Mullender. I am able to install it. ====== Doing sudo yum install /usr/bin/stethoscope should do it. That will install the package MonetDB-client-tools. On 09/03/2020 16.11, Suresh Devarakonda wrote: > Hi, > > > > I would like to use the stethoscope tool to monitor the performance of > queries. But, I do not find the tool in the MonetDB software that I > installed on my server. Can you help me to get the stethoscope tool. > > > > [root at xxxxxx<https://www.monetdb.org/mailman/listinfo/users-list> ~]# yum list installed |grep -i monet > > MonetDB.x86_64 11.35.9-20191218.el7 @monetdb > > MonetDB-SQL-server5.x86_64 11.35.9-20191218.el7 @monetdb > > MonetDB-client.x86_64 11.35.9-20191218.el7 @monetdb > > MonetDB-client-odbc.x86_64 11.35.9-20191218.el7 @monetdb > > MonetDB-release-epel.noarch 1.3-1.monetdb installed > > MonetDB-stream.x86_64 11.35.9-20191218.el7 @monetdb > > MonetDB5-server.x86_64 11.35.9-20191218.el7 @monetdb > > > > > > > > *Suresh Devarakonda* > > *84.51* <http://www.8451.com/>*° * > > *TDO - Database Support Team* > > *D:*513.632.3692 > > *M*: 859.512.1839 > > *A: *100 W. 5^th Street, Cincinnati, Ohio 45202 > > > > Cincinnati *|* Chicago *|* New York *|* Portland > > Facebook > <https://www.facebook.com/8451group/?ref=aymt_homepage_panel> LinkedIn > <https://www.linkedin.com/company/84-51/?trk=biz-companies-cym> Instagram > <https://www.instagram.com/84.51/> Twitter > <https://twitter.com/8451group> > > Please consider the environment before printing this email. > > > > > > > _______________________________________________ > users-list mailing list > users-list at monetdb.org<https://www.monetdb.org/mailman/listinfo/users-list> > https://www.monetdb.org/mailman/listinfo/users-list > ________________________________ * Previous message (by thread): stethoscope tool<https://www.monetdb.org/pipermail/users-list/2020-March/010593.html> * Messages sorted by: [ date ]<https://www.monetdb.org/pipermail/users-list/2020-March/date.html#10594> [ thread ]<https://www.monetdb.org/pipermail/users-list/2020-March/thread.html#10594> [ subject ]<https://www.monetdb.org/pipermail/users-list/2020-March/subject.html#10594> [ author ]<https://www.monetdb.org/pipermail/users-list/2020-March/author.html#10594>

1 0

stethoscope tool
by Suresh Devarakonda 09 Mar '20

09 Mar '20

Hi, I would like to use the stethoscope tool to monitor the performance of queries. But, I do not find the tool in the MonetDB software that I installed on my server. Can you help me to get the stethoscope tool. [root@xxxxxx ~]# yum list installed |grep -i monet MonetDB.x86_64 11.35.9-20191218.el7 @monetdb MonetDB-SQL-server5.x86_64 11.35.9-20191218.el7 @monetdb MonetDB-client.x86_64 11.35.9-20191218.el7 @monetdb MonetDB-client-odbc.x86_64 11.35.9-20191218.el7 @monetdb MonetDB-release-epel.noarch 1.3-1.monetdb installed MonetDB-stream.x86_64 11.35.9-20191218.el7 @monetdb MonetDB5-server.x86_64 11.35.9-20191218.el7 @monetdb [cid:image001.jpg@01D5F602.57C58E70] Suresh Devarakonda 84.51<http://www.8451.com/>° TDO - Database Support Team D: 513.632.3692 M: 859.512.1839 A: 100 W. 5th Street, Cincinnati, Ohio 45202 Cincinnati | Chicago | New York | Portland Facebook<https://www.facebook.com/8451group/?ref=aymt_homepage_panel> LinkedIn<https://www.linkedin.com/company/84-51/?trk=biz-companies-cym> Instagram<https://www.instagram.com/84.51/> Twitter<https://twitter.com/8451group> Please consider the environment before printing this email.

2 1

SP2 R integration
by ALEX OSSIPOV 22 Feb '20

22 Feb '20

I am not able to get R part working in latest SP2 and getting this error: LoaderException:loadLibrary:Loading error failed to open lib rary rapi (from within file '/usr/lib64/monetdb5/lib_rapi.so'): libR.so: cannot open shared object file: No such file or directory I use Centos 7 package MonetDB-R.x86_64_11.35.15-20200217.el7 and wonder if it was build with --enable-R-shlib option. SP1 did not have this problem Thanks Alex

2 2

Announcement: New Nov2019-SP2 Bugfix release of MonetDB suite
by Sjoerd Mullender 21 Feb '20

21 Feb '20

The MonetDB team at MonetDB BV is pleased to announce the Nov2019-SP2 bugfix release of the MonetDB suite of programs. More information about MonetDB can be found on our website at <https://www.monetdb.org/>. For details on this release, please see the release notes at <https://www.monetdb.org/Downloads/ReleaseNotes>. As usual, the download location is <https://www.monetdb.org/downloads/>. Nov 2019-SP2 bugfix release (11.35.15) Bug Fixes * 6805: Using the cascade operator in a drop table statement ends in an exit from the Monetdb shell. * 6807: Median_avg and quantile_avg ignore NULL values * 6815: query with ifthenelse() crashes mserver5 * 6816: Monetdb Crashes on INSERT statement after ALTER statement in another connection * 6817: running analyze on a schema which contains a stream table stops with an error * 6819: functions do not persist

1 0

dropped functions
by ALEX OSSIPOV 18 Feb '20

18 Feb '20

I noticed every time Monetdb service is bounced all created functions disappear from database. Is it a bug or a way functions are handled in MonetDB? Thanks Alex.

2 1

search links are broken
by ALEX OSSIPOV 17 Feb '20

17 Feb '20

All links in Search results point to localhost:8080 instead of site. Can it be fixed? Thanks Alex

2 1

Include additional libraries
by Oxana Dolmatova 13 Feb '20

13 Feb '20

Hello, Could you tell what would be the easiest way to include additional libraries into the compilation process of MonetDB? More specifically I'm interested in calling MKL operations inside BAT Algebra operations. For example, mkl standalone operation of QR decomposition (in file mkl_qr.c all the additional headers are included) is compiled as follows: gcc -c mkl_qr.c gcc -o mkl_qr mkl_qr.o -L${MKLROOT}/lib/intel64 -Wl,--no-as-needed -lmkl_intel_ilp64 -lmkl_gnu_thread -lmkl_core -lgomp -lpthread -lm -ldl What would be the proper Makefile to edit and extend with additional libraries if I want to use MKL in batcalc.c, for example? Thank you and best regards, Oksana

1 0

How can i change plan choice according new constraint
by DEMBELE Simon Pierre 10 Feb '20

10 Feb '20

hello, I want to know if it is possible to change how monetdb choose a plan by adding new contraints. -- Simon

2 2