File APIs for Java Developers
Manipulate DOC, XLS, PPT, PDF and many others from your application.
http://aspose.com/file-tools
The moose likes Performance and the fly likes Too many files open Big Moose Saloon
  Search | Java FAQ | Recent Topics | Flagged Topics | Hot Topics | Zero Replies
Register / Login
JavaRanch » Java Forums » Java » Performance
Bookmark "Too many files open" Watch "Too many files open" New topic
Author

Too many files open

Satyavrat Bondre
Greenhorn

Joined: Sep 28, 2011
Posts: 6
Hello Everyone,
I just signed up at the ranch. Not sure where to put this topic so I kind of put it here.

I have a function that is supposed to solve something recursively.

if i may give you the code structure :

for loop for n
for loop for t
call function f(n,t)


function f
get required values for n,t from database
do some calculations
insert into data base
end function

the calculations for n and t depend on the same previous calculations for n and t (namely n from 0 to n and t-1). this is taking too long(obviously because of all the database calls). I tried converting the entire thing into an array and passing it around but i get a too many files open error. I double/triple checked but i have no files open. i dont even open any file, leave alone forget to close it.


Any help on this would be much appreciated. Also, sorry if this is the wrong forum, wasn't sure where to put it.

Amit J Thakur
Greenhorn

Joined: Sep 26, 2011
Posts: 13
Not sure what are you intending to do. But why cant you do what you are doing on the database server, instead of the application. The kind of case that you have mentioned seems to be a good candidate for running on the database server itself.
Satyavrat Bondre
Greenhorn

Joined: Sep 28, 2011
Posts: 6
Hi Amit,

Thanks for your reply. The thing with the database method is that, it slows down the entire process a lot since I have to keep inserting the values and then query them again. A future calculation depends on the calculation being done right now, but both are referenced as independent values in another part of the application.
Amit J Thakur
Greenhorn

Joined: Sep 26, 2011
Posts: 13
It is still not very clear to me. But if you have to do a lot of inserts it is always to do it on the server side, if possible.

With whatever I have understood so far, I think you are reading some values from your DB, doing some processing and then inserting it back on to the DB. This could be easily achieved on the database server (depending on what calculations you want to do and the DBMS system being used). Getting data to your application, processing it and inserting it back may consume more resources overall. If you have a good configuration on your database server, I would prefer it doing it there itself, rather than doing an IO.
Satyavrat Bondre
Greenhorn

Joined: Sep 28, 2011
Posts: 6
the inserts are fine, the problem is the values i insert are used later while calculating a value for another insert

basically

F(n,t) depends on F(a,b) where a<n.

I insert the calculated value of F(a,b) as and when I do the calculation but when I need to calculate F(n,t), I end up querying the database for F(a,b).
Winston Gutkowski
Bartender

Joined: Mar 17, 2011
Posts: 7700
    
  20

the inserts are fine, the problem is the values i insert are used later while calculating a value for another insert
basically F(n,t) depends on F(a,b) where a<n.

I insert the calculated value of F(a,b) as and when I do the calculation but when I need to calculate F(n,t), I end up querying the database for F(a,b).

Seems to me that, unless this process is single-threaded (which seems highly unlikely if a database is involved), you have no choice but to go back to the db to get F(n, t); but it's hard to be sure without further explanation.

My suggestion: back up and explain what you're trying to do in plain English. That'll help us put it in context.

Winston
Jayesh A Lalwani
Bartender

Joined: Jan 17, 2008
Posts: 2342
    
  28

Can you post a code snippet? Being able to see the code might help. Normally just storing data in arrays shouldn't incur any IO. Maybe there's something else that you are doing that causes files to open

Have you tried taking a heap dump right before you get "Too many open files" and analyzing it? It might show you objects that are stating around in memory.
Amit J Thakur
Greenhorn

Joined: Sep 26, 2011
Posts: 13
Yes it would be better if you could explain it in plain english what exactly are you trying to do. Or just have the code snippet pasted here, if there are no Legal issues .
The information you have provided leaves a lot of room for different interpretations.


Seems to me that, unless this process is single-threaded (which seems highly unlikely if a database is involved), you have no choice but to go back to the db to get F(n, t); but it's hard to be sure without further explanation.

Winston, am not clear what you mean by single-threaded. My understanding is that even if it is multi-threaded, we can have a separate connection for each thread(Again approach would depend on the no. of max thread ever possible). Am I missing something?

Normally just storing data in arrays shouldn't incur any IO.

I meant Network IO and not disk IO. My bad, I should have been more clear. I was concerned with the way looping is done. If the value of n and t is big, that would mean going over the network time and again(again an assumption unless the application and the DB are not on the same machine), which is what I wanted to avoid.


Winston Gutkowski
Bartender

Joined: Mar 17, 2011
Posts: 7700
    
  20

Winston, am not clear what you mean by single-threaded. My understanding is that even if it is multi-threaded, we can have a separate connection for each thread(Again approach would depend on the no. of max thread ever possible). Am I missing something?

Well, from OP's initial post, I'm assuming that the F(a, b) method is reading something from the db, calculating new values and then writing them back. In a single-threaded situation, s/he could simply cache the calculated values in memory for use whenever needed; but in a multi-threaded one there could be many users trying to execute F(a, b) at the same time. That suggests to me:
1. F(a, b) needs to be done in a transaction.
2. Caching is pointless, since the calculated value could be out-of-date by the time it's used.

But maybe it's me that's missing something .

Winston
Bryan Grant
Greenhorn

Joined: Nov 04, 2011
Posts: 1
What OS are you running this on? I have seen this exception many times on AIX relating to assigned user limits (ulimit -a) being exceeded. If this sounds similar, try setting the ulimits to unlimited and work back to determine which setting is being exceeded.
 
I agree. Here's the link: http://aspose.com/file-tools
 
subject: Too many files open