wood burning stoves*
The moose likes Performance and the fly likes Reading a large JSON file in Java Big Moose Saloon
  Search | Java FAQ | Recent Topics | Flagged Topics | Hot Topics | Zero Replies
Register / Login


Win a copy of Murach's Java Servlets and JSP this week in the Servlets forum!
JavaRanch » Java Forums » Java » Performance
Reply locked New topic
Author

Reading a large JSON file in Java

Tanishka chandrasekaran
Greenhorn

Joined: Oct 22, 2013
Posts: 14
Hi ,
Iam new to JSON and i need help in reading a large json file which is really huge and it contains pair of id and URL in a array as shown below.. i need to get this all id and URL from this file in my java code and need to perform operations . Can someone give me suggestions on how to read such a big file and store the contents in java code to process it further.
Ulf Dittmer
Marshal

Joined: Mar 22, 2005
Posts: 41029
    
  43
Is the question about how to consume JSON in general, or about a situation where the general approach is not fast enough due to data volume? If the latter, is it an actual problem, or do you fear it could become one?


Ping & DNS - my free Android networking tools app
Tanishka chandrasekaran
Greenhorn

Joined: Oct 22, 2013
Posts: 14
Right now iam facing a memory issue due to reading a large json file from the url. Basically the file size is close to 90 mb .
I need to read this json file which just has one array WITH many ID,URL pairs in it .. I need to read each id and based on each id ,there is a file operation.
So i wanted to begin my code from scratch as something went wrong there. So i wanted in general the efficient way of parsing such a huge file and looping through each line for this ID and URL.
Frank Pavageau
Greenhorn

Joined: Jul 20, 2010
Posts: 10
Use a JSON library providing a streaming mode, such as Jackson.
fred rosenberger
lowercase baba
Bartender

Joined: Oct 02, 2003
Posts: 11145
    
  16

is this question really different from this one? If not, please don't make duplicates, as it wastes everyone's time.


There are only two hard things in computer science: cache invalidation, naming things, and off-by-one errors
Tanishka chandrasekaran
Greenhorn

Joined: Oct 22, 2013
Posts: 14
i have changed my code something like this:

My json file have to be read from a url through http . once i read the url ,open the url stream and i have to read line by line . its a json format file . Please help.
i have tried to read from the url as below:

InputStream is = new URL(url).openStream();
BufferedReader reader = new BufferedReader(new InputStreamReader(
is, Charset.forName("UTF-8")));
String line, results = "";
while ((line = reader.readLine()) != null) {
results += line;
}
reader.close();
is.close();
JSONObject json = new JSONObject(results);
JSONArray fileArray = json.getJSONArray("Documents");

then iam looping the Array again for each line to perform some actions.Any suggestions to improve the code here.
fred rosenberger
lowercase baba
Bartender

Joined: Oct 02, 2003
Posts: 11145
    
  16

You haven't said how this thread differs from the other, so I am closing this as a duplicate. Any follow-up posts should be in the other thread.
 
I agree. Here's the link: http://aspose.com/file-tools
 
subject: Reading a large JSON file in Java
 
Similar Threads
REST tutorial
Getting message as Invalid JSON
passing response as json vs. string array
JSON in Java
reading a file from a url