aspose file tools*
The moose likes JDBC and the fly likes MySQL uses too little memory Big Moose Saloon
  Search | Java FAQ | Recent Topics | Flagged Topics | Hot Topics | Zero Replies
Register / Login
JavaRanch » Java Forums » Databases » JDBC
Bookmark "MySQL uses too little memory" Watch "MySQL uses too little memory" New topic
Author

MySQL uses too little memory

Scott Selikoff
author
Saloon Keeper

Joined: Oct 23, 2005
Posts: 3716
    
    5

Hi All,

I'm running through some bulk loading SQL procedures that are taking hours to run. It might be they need to take hours, but one thing I've noticed is that MySQL never requests much memory (~50 megs) throughout the process.

I know there's a dozen or so variables to increase buffer sizes, and I've tried a lot of them with little to no change. Any suggested settings I should look at?

-Scott


My Blog: Down Home Country Coding with Scott Selikoff
Bauke Scholtz
Ranch Hand

Joined: Oct 08, 2006
Posts: 2458
MySQL ships with several default configuration files. Start here: http://dev.mysql.com/doc/refman/5.1/en/option-files.html#option-files-preconfigured
Scott Selikoff
author
Saloon Keeper

Joined: Oct 23, 2005
Posts: 3716
    
    5

Thanks Bauke, but I was looking for specific tips from people who have seen issues like this before. I've read through the MySQL documentation and its really not that helpful.
Bauke Scholtz
Ranch Hand

Joined: Oct 08, 2006
Posts: 2458
So you have already tried using for example the my-huge.cnf instead? If that didnĀ“t utilize more memory, then the problem may lie somewhere else or there is actually not a problem at all.

At least, the variables are explained in detail here: http://dev.mysql.com/doc/refman/5.1/en/server-system-variables.html Most important are those variables with "buffer" in it. For example key_buffer_size and read_buffer_size.
Scott Selikoff
author
Saloon Keeper

Joined: Oct 23, 2005
Posts: 3716
    
    5

Nope it does not. I'm bulk loading a couple gigs of data, tried using INSERT and LOAD DATA INFILE, but it seems to grind after a certain point. It is a single threaded execution though,

Again, I don't need a list of system variables, there's too many as it is and I've read through everything you've sent all ready. I'm just trying to find out if anyone has seen cases where mysql's memory size stays so small and specific ways to improve it.
Scott Selikoff
author
Saloon Keeper

Joined: Oct 23, 2005
Posts: 3716
    
    5

Found the variable to increase memory size for INNODB systems: innodb_buffer_pool_size
It was set at ~10 megs... increasing it to 800 megs definitely increased memory size. Not sure yet about performance though.
Scott Selikoff
author
Saloon Keeper

Joined: Oct 23, 2005
Posts: 3716
    
    5

Just as an update, changing the InnoDB definitely fixed the issue. As a benchmark, one bulk load went from 4 hours to 15 minutes. Definitely an vital setting to set if you want MySQL to really use the memory it has available.
 
I agree. Here's the link: http://aspose.com/file-tools
 
subject: MySQL uses too little memory