Neil Barton

Ranch Hand
+ Follow
since Aug 24, 2011
Cows and Likes
Cows
Total received
0
In last 30 days
0
Total given
0
Likes
Total received
2
Received in last 30 days
0
Total given
0
Given in last 30 days
0
Forums and Threads
Scavenger Hunt
expand Ranch Hand Scavenger Hunt
expand Greenhorn Scavenger Hunt

Recent posts by Neil Barton

Follow up: I noticed the CPU was only at 25% on NLB8 so I increased the number of threads in my app to 22 (after some trial and error), this has got the CPU and Disk to 100%. NLB7 is running with 6 threads. The number of Selects per second is now about 70 but it is still nowhere near as fast as the NLB7 although the spec is vastly better. It seems to be something to do with the configuration of MySQL but I'm not sure where to start with that. I had a look at the MySQL System variables on each machine. The results are attached in the VarDiffs file.
Any advice would be appreciated.
VariableNLB7NLB8
host_cache_size628179
innodb_buffer_pool_chunk_size1342177288388608
innodb_buffer_pool_size1342177288388608
innodb_log_buffer_size167772161048576
innodb_open_files4000300
innodb_thread_concurrency033
innodb_thread_sleep_delay100000
innodb_version8.0.138.0.20
max_allowed_packet671088644194304
max_connections512151
myisam_max_sort_file_size2146435072107374182400
myisam_sort_buffer_size8388608512753664
open_files_limit105706209
performance_schema_error_size44684697
read_buffer_size13107265536
table_definition_cache20001400
table_open_cache40002000
thread_cache_size1310
tmp_table_size16777216262144000
version8.0.138.0.20

It looks like it's throttling threads for some reason but overall it looks like I've configured it as a small server rather than one which handles mutliple threads, is there some sort of system wide setting that I need to change? Or was there a decision I got wrong when installing? It is an innodb.
I tried adding innodb_thread_concurrency=0 to my.ini but that made no difference and the server still reports that it is 33! Reading around the internet there are suggestions that this is only for Solaris systems and it should be 0, but how do I set it to 0?
Further update: after some messing around I set innodb_thread_concurrency to 0 and innodb_buffer_pool_size to 26000000000 (80% of the 32gb or ram). I got the processing up to about 2400 statements per second! I did this by opening a MySQL session and changing the variables using SET GLOBAL innodb_thread_concurrency=0. But when I restart MySQL I lose the two settings. Can I put these changes into my.ini so that they are always in place? I have tried several syntax variations but nothing makes any difference. I have also tried using 'persist' in the setting statement but it doesn't accept it.
It also still says in MySQL Workbench that key efficiency is 0. I know it isn't but can I get this to update on the screen somehow?
I have 2 machines, NLB7 is a core i7 with SSD drives. I also have NLB8 which is a core i9 with M2 drives. NLB8 has always been faster (unsurprisingly). Suddenly something has changed and I'm not sure what. I had some crashes on NLB8 and upgraded the MySQL server to the latest release, that took a long time! Now when I run my application (which makes heavy use of the DB) it runs about 49 SQL statements per second and is very slow. The older NLB7 which should be slower runs 575 statements per second. What has happened to NLB8 to make it suddenly run so slow. In an effort to speed it up I have updated the BIOS, INF and ME drivers on the motherboard but nothing makes any difference. I have also tried changing the innodb_buffer_pool_size to 26000000000 but that doesn't have any impact either (the machine has 32gb of ram). What can I look at to help tune the actual DB itself, it runs fine on NLB7 so it's not the SQL.
Thanks Again Claude,
I found some code using the hikari library and got it working. Whilst implementing it I tidied up the DB access code and closed the leakage which became fairly obvious! The code now runs about 3 times as fast and that's before I try different numbers of connections to optimise it. Excellent, thanks!
Thanks Claude, I had been looking at c3p0 but haven't got very far as all the examples are in Maven. I'll have a look at Hikary, I'd seen it around but had favoured c3p0 by chance.
I have been getting some errors on my program relating to running out of sockets. In searching for a solution to this I have seen Connection Pooling as a likely solution. It also promises performance benefits and greater control over the DB usage. I'm keen to give it a go. All of the examples I could find relating to this used Eclipse, Maven or some commercial driver. I don't use any of these, I write my programs in Netbeans and that keeps things simple.

I found this site: https://docs.oracle.com/database/121/JJUCP/manage.htm#JJUCP8180 and put together the code below. It compiles but when I run it I get the error at the bottom. Can anyone tell me where I can find a simple example of connection pooling not using Eclipse, Maven or anything else? Alternatively what did I miss in my code and how do I then use this DataPool when it is working?


run:
Exception in thread "main" java.lang.NoClassDefFoundError: oracle/jdbc/OracleShardingKeyBuilder
at oracle.ucp.jdbc.PoolDataSourceFactory.getPoolDataSource(PoolDataSourceFactory.java:48)
at datapool.DataPool.main(DataPool.java:31)
Caused by: java.lang.ClassNotFoundException: oracle.jdbc.OracleShardingKeyBuilder
at java.net.URLClassLoader.findClass(URLClassLoader.java:381)
at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:338)
at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
... 2 more
C:\Users\NeilB\AppData\Local\NetBeans\Cache\8.2\executor-snippets\run.xml:53: Java returned: 1
BUILD FAILED (total time: 0 seconds)
Wow, thanks for that. It'll take me a couple of days to digest this, I've not used streams like that before. I'll work through it so I can understand it properly then I can translate it into my code. Brilliant, thanks!
3 months ago
You're right! As I said it's a bit of a contrived example. The question is how can you have one complex routine that updates different parts of an array. When you have to type This ArrayList.{something} you make it only capable of updating that variable so if you want to update a different variable you have to have a sperate procedure, unless you pass it in (as I have done in my solution) or there's a way of passing a column of an arraylist and passing a point to which part of the arraylist needs updating (which I think is the essence of the question).
3 months ago
Yes, it's a bit of a contrived example, I was trying to keep it simple. Essentially it's trying to convey 1. Run the piece of code with all the red entries, 2. Run the same piece of code with all the green entries. What's changing is the data bit on the end of the SQL. I agree, my naming conventions are all over the place and I wish I was more disciplined, it does cause me problems!
My solution seems to be working but it's as elegant as a hippo in wellingtons so if anyone can suggest a better solution I'd be very interested!
3 months ago
Thanks Junilu, That's very helpful.
I don't have to worry about security as this is a completely internal DB and will never be open to attack.
You are correct the "Red in (…)" is the syntax I want use except that some of the variables are ranges so it would come out like :
"Red IN (BEWEEN 3 and 5, BETWEEN 6 and 10, 7)" which I haven't tried but I think the compiler would reject.
The solution I've come up with is to pass another arraylist to the routine with all the values in it. It's a bit scrappy though :-(

3 months ago
I have an arraylist with many elements. I call a complex, frequently changing routine and want it to use that arraylist but I want to call it to work on different elements of the array each time. I can build the SQL before calling the routine but at some point I have to identify the element of the array I am calling and that means I have to have a different routine for each element. There are about 25 and the repeating code is quite an overhead to maintain.
In the example you can see the simple way to do it involving one routine (ComplexProcedure) for each element. But I want one routine that I can pass the values to,  I want to process all the red values and then call the same routine to process all the green values. But how do I pass that to the routine? I think I have to build a separate list each time but that seems an enormous overhead!
3 months ago
Thanks Tim,

Good to know I'm using the right thing. I'm having trouble making it work. Is there a simple guide anywhere? I use NetBeans as my environment but I can't seem to find all the JAR files it wants, it currently wants a ContactsService library that I can't seem to find anywhere. Have I missed a library?
5 months ago
I want to get contact data from Google contacts. I have looked at the google documentation, it says the GDATA libraries are now out of date and deprecated but when I start following the links for the API client they lead back to the GDATA site. I've seen a couple of questions on this site about this but they all seem to refer to the GDATA as well. Is it deprecated? In which case what has replaced it? Or is it still OK to use?
5 months ago
0.9.4 gives this error:
com.pff.PSTException: Unable to find node: 0 is desc: true
In exactly the same place as the previous error was with the earlier version so it looks like they did something but it didn't fix the problem.

Any suggestions welcome !
Nope, I still can't make it work and I can't see why not. This is my code:

It seems to get to the end of one of the folders and then tries to 'get' the next email which fails (understandably) with index out of bounds:
|  |  |  |- Email Subject: CloudSec Europe 2019: We see you clicked but did not register. Can we help?
|  |  |  |- Email Subject: ESET UK - Upcoming Encryption Training 2019
java.lang.IndexOutOfBoundsException: Index: 1478, Size: 1478
at java.util.LinkedList.checkElementIndex(LinkedList.java:555)
at java.util.LinkedList.get(LinkedList.java:476)
at com.pff.PSTFolder.getNextChild(PSTFolder.java:326)
at readpstfiles.ReadPSTFiles.processFolder(ReadPSTFiles.java:44)
at readpstfiles.ReadPSTFiles.processFolder(ReadPSTFiles.java:33)
at readpstfiles.ReadPSTFiles.processFolder(ReadPSTFiles.java:33)
at readpstfiles.ReadPSTFiles.processFolder(ReadPSTFiles.java:33)
at readpstfiles.ReadPSTFiles.main(ReadPSTFiles.java:63)

Surely the 'getnextchild' should return null if it's reached the end of the list, which would trigger the end of the loop.
Can anyone see what I'm doing wrong? I tried doing it as an iterator (thinking it would avoid the issue) but I can't get that to work either :-(
Thanks Tim,

That really is above and beyond the call of duty, much appreciated. 0.9.5 crashes out with an index out of bounds error but that's another debugging game, you have got me past the one I was stumped on.

Thanks gain,
Neil