aspose file tools*
The moose likes Performance and the fly likes Reading a large JSON file in Java Big Moose Saloon
  Search | Java FAQ | Recent Topics | Flagged Topics | Hot Topics | Zero Replies
Register / Login
JavaRanch » Java Forums » Java » Performance
Reply locked New topic
Author

Reading a large JSON file in Java

Tanishka chandrasekaran
Greenhorn

Joined: Oct 22, 2013
Posts: 14
Hi ,
Iam new to JSON and i need help in reading a large json file which is really huge and it contains pair of id and URL in a array as shown below.. i need to get this all id and URL from this file in my java code and need to perform operations . Can someone give me suggestions on how to read such a big file and store the contents in java code to process it further.
Ulf Dittmer
Marshal

Joined: Mar 22, 2005
Posts: 42950
    
  71
Is the question about how to consume JSON in general, or about a situation where the general approach is not fast enough due to data volume? If the latter, is it an actual problem, or do you fear it could become one?
Tanishka chandrasekaran
Greenhorn

Joined: Oct 22, 2013
Posts: 14
Right now iam facing a memory issue due to reading a large json file from the url. Basically the file size is close to 90 mb .
I need to read this json file which just has one array WITH many ID,URL pairs in it .. I need to read each id and based on each id ,there is a file operation.
So i wanted to begin my code from scratch as something went wrong there. So i wanted in general the efficient way of parsing such a huge file and looping through each line for this ID and URL.
Frank Pavageau
Greenhorn

Joined: Jul 20, 2010
Posts: 10
Use a JSON library providing a streaming mode, such as Jackson.
fred rosenberger
lowercase baba
Bartender

Joined: Oct 02, 2003
Posts: 11499
    
  16

is this question really different from this one? If not, please don't make duplicates, as it wastes everyone's time.


There are only two hard things in computer science: cache invalidation, naming things, and off-by-one errors
Tanishka chandrasekaran
Greenhorn

Joined: Oct 22, 2013
Posts: 14
i have changed my code something like this:

My json file have to be read from a url through http . once i read the url ,open the url stream and i have to read line by line . its a json format file . Please help.
i have tried to read from the url as below:

InputStream is = new URL(url).openStream();
BufferedReader reader = new BufferedReader(new InputStreamReader(
is, Charset.forName("UTF-8")));
String line, results = "";
while ((line = reader.readLine()) != null) {
results += line;
}
reader.close();
is.close();
JSONObject json = new JSONObject(results);
JSONArray fileArray = json.getJSONArray("Documents");

then iam looping the Array again for each line to perform some actions.Any suggestions to improve the code here.
fred rosenberger
lowercase baba
Bartender

Joined: Oct 02, 2003
Posts: 11499
    
  16

You haven't said how this thread differs from the other, so I am closing this as a duplicate. Any follow-up posts should be in the other thread.
 
I agree. Here's the link: http://aspose.com/file-tools
 
subject: Reading a large JSON file in Java