• Post Reply Bookmark Topic Watch Topic
  • New Topic
programming forums Java Mobile Certification Databases Caching Books Engineering Micro Controllers OS Languages Paradigms IDEs Build Tools Frameworks Application Servers Open Source This Site Careers Other Pie Elite all forums
this forum made possible by our volunteer staff, including ...
Marshals:
  • Campbell Ritchie
  • Jeanne Boyarsky
  • Ron McLeod
  • Paul Clapham
  • Liutauras Vilda
Sheriffs:
  • paul wheaton
  • Rob Spoor
  • Devaka Cooray
Saloon Keepers:
  • Stephan van Hulst
  • Tim Holloway
  • Carey Brown
  • Frits Walraven
  • Tim Moores
Bartenders:
  • Mikalai Zaikin

Reading file splitting data into individual readings

 
Ranch Hand
Posts: 76
  • Mark post as helpful
  • send pies
    Number of slices to send:
    Optional 'thank-you' note:
  • Quote
  • Report post to moderator
Hi everybody.

I have a list of weather for a city from 1908 till 2013. I am trying to write a program that reads the reading, writes it into an array and gives the average rainfall per year. like the image below


I have problem splitting readings by year. I was gonna use file.reaLine.split(); but I'm not sure how to do it to be able to split data by year.
Any idea?
Thank you
 
Sheriff
Posts: 7125
184
Eclipse IDE Postgres Database VI Editor Chrome Java Ubuntu
  • Likes 2
  • Mark post as helpful
  • send pies
    Number of slices to send:
    Optional 'thank-you' note:
  • Quote
  • Report post to moderator
Do you know Regular Expressions (regex)? They can be complex but they are designed for this kind of stuff. Look at java.util.regex.Pattern.
 
Bartender
Posts: 5465
212
  • Likes 5
  • Mark post as helpful
  • send pies
    Number of slices to send:
    Optional 'thank-you' note:
  • Quote
  • Report post to moderator
Isn't it very much easier to open this file in excel or libreoffice,
and derive the average rainfall with a couple of mouseclicks?
 
saeid jamali
Ranch Hand
Posts: 76
  • Mark post as helpful
  • send pies
    Number of slices to send:
    Optional 'thank-you' note:
  • Quote
  • Report post to moderator

Knute Snortum wrote:Do you know Regular Expressions (regex)? They can be complex but they are designed for this kind of stuff. Look at java.util.regex.Pattern.



Do you know anything easier? I'm newbie
 
saeid jamali
Ranch Hand
Posts: 76
  • Mark post as helpful
  • send pies
    Number of slices to send:
    Optional 'thank-you' note:
  • Quote
  • Report post to moderator

Piet Souris wrote:Isn't it very much easier to open this file in excel or libreoffice,
and derive the average rainfall with a couple of mouseclicks?



you are right but it's my assignment! (
 
Bartender
Posts: 1845
10
  • Likes 1
  • Mark post as helpful
  • send pies
    Number of slices to send:
    Optional 'thank-you' note:
  • Quote
  • Report post to moderator
Ok, so what are you going to store in this array?
An object/class?
A number?
In your design, what does each entry in the array contain?

Reading one line as a string and then using the split() method makes sense to me.
That would give you an array of Strings, which presumable the item at index 0 would be the year, index 1 the month etc etc. That entirely depends upon the format of the file.
Is this screenshot an exact example of what the file looks like or a conceptual one? I don't see any rainfall data in it is all :-)

 
Saloon Keeper
Posts: 10687
85
Eclipse IDE Firefox Browser MySQL Database VI Editor Java Windows ChatGPT
  • Mark post as helpful
  • send pies
    Number of slices to send:
    Optional 'thank-you' note:
  • Quote
  • Report post to moderator
Your posted image of the data is too small for me to read. Could you cut and paste the first 10 lines of the data file here. Also nice to have would be a cut and paste of 10 data file lines where the year changes.
 
Piet Souris
Bartender
Posts: 5465
212
  • Likes 2
  • Mark post as helpful
  • send pies
    Number of slices to send:
    Optional 'thank-you' note:
  • Quote
  • Report post to moderator
I see; no excel/libreoffice then

Well, what Knute suggested would be fine indeed, but a regex might not
be optimal for a newbie.

Next, a Scanner comes in handy. Do you know these? Skip the first
four lines, and then use methods like scnner.nectInt, scanner.nextDouble
and so on.

You could create a class, where every instance represents one observation.
It would then be a doddle to derive means with Java8 streams.

If that is a little futuristic:
I would use a 2D array. First dimension for the year, second dimension
for the month, value will be the rainfall. Think of how you would store
1908, month 1 into array[0][0].

Then, getting a year average should be straightforward.
 
Piet Souris
Bartender
Posts: 5465
212
  • Mark post as helpful
  • send pies
    Number of slices to send:
    Optional 'thank-you' note:
  • Quote
  • Report post to moderator

Stefan Evans wrote:(...)
Is this screenshot an exact example of what the file looks like or a conceptual one? I don't see any rainfall data in it is all :-)


Missed that! Well, assuming "---" means "nothing at all", that certainly makes
calculating the average a tad easier
 
saeid jamali
Ranch Hand
Posts: 76
  • Mark post as helpful
  • send pies
    Number of slices to send:
    Optional 'thank-you' note:
  • Quote
  • Report post to moderator
So the user is gonna enter the name of the city (I have 5 cities) and the average rainfall, sunny hours and temperature BY YEAR comes up (for now I'm focusing on one which is rainfall for each year). I'm gonna make a 2D array table for each city with the number of rows and 7 columns as I have 7 columns in every data list and variable rows that goes up to 1257 rows. My biggest problem for now is the years start from 1905 till 2013 so if somehow I can split data by year and write it into a 2D array for example all of the data for 1905, 1906,1907 and son on are split.
I think I'm doing something wrong and there should be a way to calculate the average for every year because the program must work with any file with the same format.

I upload another image for you guys.

https://www.imageupload.co.uk/image/cVDF
 
Carey Brown
Saloon Keeper
Posts: 10687
85
Eclipse IDE Firefox Browser MySQL Database VI Editor Java Windows ChatGPT
  • Likes 1
  • Mark post as helpful
  • send pies
    Number of slices to send:
    Optional 'thank-you' note:
  • Quote
  • Report post to moderator
 
Stefan Evans
Bartender
Posts: 1845
10
  • Likes 1
  • Mark post as helpful
  • send pies
    Number of slices to send:
    Optional 'thank-you' note:
  • Quote
  • Report post to moderator
So they point Carey was making with that code sample is

Step 1: Read the data from the file into a data structure of some sort. In this case, he is suggesting a List of "MonthlyRecord" objects, where each monthly record object stores the data from one line in the input file.
Step 2: Analyze the data in memory. e.g. loop through all the data, check if that record pertains to the year of interest, and get its rainfall value (if present)


I'm gonna make a 2D array table for each city with the number of rows and 7 columns as I have 7 columns in every data list and variable rows that goes up to 1257 rows


So I would assume that would be something like


Have you learned about Objects yet?
Personally I would prefer to have a MonthlyRecord[]

Using a List instead of an Array makes things easier, because you don't need to know how large the List is before you create it, you can just add things to it.
 
saeid jamali
Ranch Hand
Posts: 76
  • Mark post as helpful
  • send pies
    Number of slices to send:
    Optional 'thank-you' note:
  • Quote
  • Report post to moderator

Carey Brown wrote:



I really like the idea of arraylist and objects. I have been reading about them so I can do my project this way. I'm gonna give it a try to see if I can do it this way.
Can anybody please them me what is happening in the for each loop(I assume we need the for-each loop for reading the data into araylist? )? what does this if statement mean and what does this mean

Thannk you very much guys
 
Carey Brown
Saloon Keeper
Posts: 10687
85
Eclipse IDE Firefox Browser MySQL Database VI Editor Java Windows ChatGPT
  • Likes 1
  • Mark post as helpful
  • send pies
    Number of slices to send:
    Optional 'thank-you' note:
  • Quote
  • Report post to moderator
Line 15 can be read as: for each record in the list of records. Each time through the loop the variable 'record' will contain the next instance of a MonthlyRecord object (not a list).
Line 17: Check to see if the year has changed from the year of the previous record.
Line 24: Save the year of this record for the purpose of checking it on line 17 on the next go around of the loop. 'Year' serves no other purpose than to detect when we have come across a new set of MonthlyRecord's for the next year.

The variable 'year' might have been more clearly written as 'previousYear'.

Line 17 might have been better written as

This would deal with the special case of first time through the loop.

The class 'MonthlyRecord' on line 34 is just a sketch showing the 'year' field. This would need to be flushed out with additional fields, one for each of the columns in your data file. You'll need a constructor that takes a single String that is a line read from the input file, and then parses it into the appropriate fields. In my example I've assumed that when you parse the year from the line of data that you'll convert it to an int.

Edit: "record.getYear()" takes a 'record' object, which in this case is an instance of the MonthlyRecord class, and invokes the class's 'getYear()' method, which in this case returns the object's field 'year'.
 
Bartender
Posts: 10780
71
Hibernate Eclipse IDE Ubuntu
  • Likes 1
  • Mark post as helpful
  • send pies
    Number of slices to send:
    Optional 'thank-you' note:
  • Quote
  • Report post to moderator

Stefan Evans wrote:So the point Carey was making with that code sample is
Step 1: Read the data from the file into a data structure of some sort. In this case, he is suggesting a List of "MonthlyRecord" objects, where each monthly record object stores the data from one line in the input file...


And I'd go even further than that. A MonthlyRecord class should hold ALL the information you need to store, viz:and furthermore, IT should be responsible for pulling it's values from a "record" (presumably a line of text), viz:and then you can add it to your list with something like:
  MonthlyRecord mr = new MonthlyRecord( myFile.readLine() );
  records.add(mr);
and then sort/extract/muck about with it all you want.

HIH

Winston
 
saeid jamali
Ranch Hand
Posts: 76
  • Mark post as helpful
  • send pies
    Number of slices to send:
    Optional 'thank-you' note:
  • Quote
  • Report post to moderator

Winston Gutkowski wrote:

Stefan Evans wrote:So the point Carey was making with that code sample is
Step 1: Read the data from the file into a data structure of some sort. In this case, he is suggesting a List of "MonthlyRecord" objects, where each monthly record object stores the data from one line in the input file...


And I'd go even further than that. A MonthlyRecord class should hold ALL the information you need to store, viz:and furthermore, IT should be responsible for pulling it's values from a "record" (presumably a line of text), viz:and then you can add it to your list with something like:
  MonthlyRecord mr = new MonthlyRecord( myFile.readLine() );
  records.add(mr);
and then sort/extract/muck about with it all you want.

HIH

Winston


Is this the way of adding a file to an array list?

Cause I tried this way but when I print out the array list with this

compiler gives an error and says index:0 size:0 I assume it means it could not add it to array list so there is nothing in there. I tried many different ways.
I tried this but it says The method add(Assignment.MonthlyRecord) in the type List<Assignment.MonthlyRecord> is not applicable for the arguments (String)

I tried a for loop but every time there is something wrong so I'm just wondering if there is something wrong with the structure of code or I'm doing it wrong.
I have one public class followed by a public static void plus these below
 
Carey Brown
Saloon Keeper
Posts: 10687
85
Eclipse IDE Firefox Browser MySQL Database VI Editor Java Windows ChatGPT
  • Likes 1
  • Mark post as helpful
  • send pies
    Number of slices to send:
    Optional 'thank-you' note:
  • Quote
  • Report post to moderator
Close, but change it to

If you want to print out all the records you'll need a toString() method in the MonthlyRecord class that formats the record the way you'd like.
Then, loop through all the records like this

Of course you'll need to skip the rows in the file that make up the header.
 
saeid jamali
Ranch Hand
Posts: 76
  • Mark post as helpful
  • send pies
    Number of slices to send:
    Optional 'thank-you' note:
  • Quote
  • Report post to moderator
What about this? does do the job?


If yes then why do I need String[] fields = record.split("\\s+");in my public class as shown above? cause without the split I was getting Format exception.
and another thing is with this structure I have to be able to print the value of the tmax like System.out.println(MonthlyRecord.tmax()); is it right? so I can print it, calculate it or do what ever with it.

Guys sorry for asking too many questions but every time you guys mention something I read about it and I'm really interested to learn java. I learnt a lot of stuff from this forum more than any classes.
 
Knute Snortum
Sheriff
Posts: 7125
184
Eclipse IDE Postgres Database VI Editor Chrome Java Ubuntu
  • Mark post as helpful
  • send pies
    Number of slices to send:
    Optional 'thank-you' note:
  • Quote
  • Report post to moderator
This line:

will loop until the end of the file. So the for loop will execute once with i = 0 and then "file" will be at the end of the file for all the other iterations. If you want to read one line, try:

and test line for null with an if statement.
 
Carey Brown
Saloon Keeper
Posts: 10687
85
Eclipse IDE Firefox Browser MySQL Database VI Editor Java Windows ChatGPT
  • Likes 1
  • Mark post as helpful
  • send pies
    Number of slices to send:
    Optional 'thank-you' note:
  • Quote
  • Report post to moderator

saeid jamali wrote:What about this? does do the job?


If yes then why do I need String[] fields = record.split("\\s+");in my public class as shown above? cause without the split I was getting Format exception.
and another thing is with this structure I have to be able to print the value of the tmax like System.out.println(MonthlyRecord.tmax()); is it right? so I can print it, calculate it or do what ever with it.

Guys sorry for asking too many questions but every time you guys mention something I read about it and I'm really interested to learn java. I learnt a lot of stuff from this forum more than any classes.


You already have a split() in your MonthlyRecord constructor, you don't need one here.
Your for loop will never execute because records.size() should be zero at this point. You don't need the for() here at all, the while() should suffice except for reading and throwing away a couple of header lines.
 
saeid jamali
Ranch Hand
Posts: 76
  • Mark post as helpful
  • send pies
    Number of slices to send:
    Optional 'thank-you' note:
  • Quote
  • Report post to moderator
Ok guys just one more question about reading file and that's it
My code looks alright to me but I'm getting NumberFormatException for some reasons and I can't figure out what's happening.
So I'm reading the file like this in my main

So I trim the leading and trailing space before adding it to my arraylist.

Then in my constructor class I have the following just before parse the records.

So I split the reading with one or more white space and then it goes through a loop and replaces all non digits with "" and I keep "." and "-" for decimal and negative numbers.
This was working fine last night and I don't remember changing anything but all of a sudden it stopped working. I'm so confused cause if there is any sign or letter attached to a number it will works fine and replaces them. but if there is any word on it's own in the header it gives exception. It says NumberFormatException strin "".

Maybe when it replaces a non digit with ("") it causes the exception?? somewhere some String being converted to int but I have no clue.
and the other problem is my no data sign is like "---" if I keep "-" for negatives number it won't be replaced and it gives me exception !!
 
Knute Snortum
Sheriff
Posts: 7125
184
Eclipse IDE Postgres Database VI Editor Chrome Java Ubuntu
  • Likes 1
  • Mark post as helpful
  • send pies
    Number of slices to send:
    Optional 'thank-you' note:
  • Quote
  • Report post to moderator
I looks like what you wanted to use was replaceAll(). It takes a regex.
 
saeid jamali
Ranch Hand
Posts: 76
  • Mark post as helpful
  • send pies
    Number of slices to send:
    Optional 'thank-you' note:
  • Quote
  • Report post to moderator

Knute Snortum wrote:I looks like what you wanted to use was replaceAll(). It takes a regex.


Yea I tried it as well I get the same exception. it says NumberFormatException "Location" which location is the first word in my file without All. and it says NumberFormatException: "" with All.
I deleted

It works without if I delete the text on the top of file and this "---" which means empty data in my file.
 
Knute Snortum
Sheriff
Posts: 7125
184
Eclipse IDE Postgres Database VI Editor Chrome Java Ubuntu
  • Likes 1
  • Mark post as helpful
  • send pies
    Number of slices to send:
    Optional 'thank-you' note:
  • Quote
  • Report post to moderator
Could you post the stack trace of the exception, the surrounding code, and the data that causes the problem?
 
Winston Gutkowski
Bartender
Posts: 10780
71
Hibernate Eclipse IDE Ubuntu
  • Likes 1
  • Mark post as helpful
  • send pies
    Number of slices to send:
    Optional 'thank-you' note:
  • Quote
  • Report post to moderator

saeid jamali wrote:Then in my constructor class I have the following just before parse the records...


First: DON'T GUESS.

What you need is to know EXACTLY what every single record in your file looks like, and what order they come in.

For example, as Carey already said, you'll probably need to make sure that your constructor ignores header records - so I'd suggest writing a
  public static final boolean isHeader(String record) { ...
method and have the constructor throw an Exception if you try to call it with a header record.

Then your main() logic might look something like:but before you can do that, you MUST know precisely what you're going to be getting.

Winston
 
Winston Gutkowski
Bartender
Posts: 10780
71
Hibernate Eclipse IDE Ubuntu
  • Likes 1
  • Mark post as helpful
  • send pies
    Number of slices to send:
    Optional 'thank-you' note:
  • Quote
  • Report post to moderator

Winston Gutkowski wrote:... I'd suggest writing a
  public static final boolean isHeader(String record) { ...
method...


Or possibly even better, a:
  public static final boolean isValidData(String record) { ...
method that checks that everything in the record is correctly formatted, eg:I leave the tests up to you.

Winston
 
saeid jamali
Ranch Hand
Posts: 76
  • Mark post as helpful
  • send pies
    Number of slices to send:
    Optional 'thank-you' note:
  • Quote
  • Report post to moderator

Winston Gutkowski wrote:

Winston Gutkowski wrote:... I'd suggest writing a
  public static final boolean isHeader(String record) { ...
method...


Or possibly even better, a:
  public static final boolean isValidData(String record) { ...
method that checks that everything in the record is correctly formatted, eg:I leave the tests up to you.

Winston


but it doesn't ignore the header which is a text does it?
So I uploaded two pictures of how the file looks from beginning to end
top.jpg
[Thumbnail for top.jpg]
but.jpg
[Thumbnail for but.jpg]
 
Marshal
Posts: 79151
377
  • Likes 1
  • Mark post as helpful
  • send pies
    Number of slices to send:
    Optional 'thank-you' note:
  • Quote
  • Report post to moderator
Please don't post pictures but text. Those pictures are illegible on my screen. But neither the text nor the pictures tell us what the correct format for the file is. Only you know that.
 
saeid jamali
Ranch Hand
Posts: 76
  • Mark post as helpful
  • send pies
    Number of slices to send:
    Optional 'thank-you' note:
  • Quote
  • Report post to moderator
 
saeid jamali
Ranch Hand
Posts: 76
  • Mark post as helpful
  • send pies
    Number of slices to send:
    Optional 'thank-you' note:
  • Quote
  • Report post to moderator
and there is white space between every column of course.
I'm trying to read only numbers and for that I used regex in my constructor so it replaces any non digits to ("") but still I'm getting exception. shouldn't it replace those words on the top and bottom with ("")?
 
Knute Snortum
Sheriff
Posts: 7125
184
Eclipse IDE Postgres Database VI Editor Chrome Java Ubuntu
  • Mark post as helpful
  • send pies
    Number of slices to send:
    Optional 'thank-you' note:
  • Quote
  • Report post to moderator
It's hard to tell what the problem is without the code you've written. Can you post your entire code here?
 
saeid jamali
Ranch Hand
Posts: 76
  • Mark post as helpful
  • send pies
    Number of slices to send:
    Optional 'thank-you' note:
  • Quote
  • Report post to moderator

Knute Snortum wrote:It's hard to tell what the problem is without the code you've written. Can you post your entire code here?


The entire code is a lot of code but the part for reading data is like this.
I read the data like this from the file.


and this is my constructor:



this code does the job. it ignores all the symbols and signs like ("@", "*", "#") except for that ("---") because I kept this sign in my replaceAll method for negative numbers this is the first problem. the second problem is that it should ignores all the alphabet as well but seems like it doesn't so I need to get rid of this letters somehow.
 
Winston Gutkowski
Bartender
Posts: 10780
71
Hibernate Eclipse IDE Ubuntu
  • Likes 1
  • Mark post as helpful
  • send pies
    Number of slices to send:
    Optional 'thank-you' note:
  • Quote
  • Report post to moderator

saeid jamali wrote:and this is my constructor:


Well that's wrong right there isn't it? In fact, practically all your conversions are.

Look at your data. HARD.

Winston
 
saeid jamali
Ranch Hand
Posts: 76
  • Mark post as helpful
  • send pies
    Number of slices to send:
    Optional 'thank-you' note:
  • Quote
  • Report post to moderator

Winston Gutkowski wrote:

saeid jamali wrote:and this is my constructor:


Well that's wrong right there isn't it? In fact, practically all your conversions are.

Look at your data. HARD.

Winston


Do you think so? are you saying this because of "---" sign? if yea this sign can be any where in the data as you can see tmax, tmin, sunny columns also have this ("---") and I have 4 more files formatted exactly the same but this "---" can be found any where. Otherwise it calculates my data perfectly.
 
Winston Gutkowski
Bartender
Posts: 10780
71
Hibernate Eclipse IDE Ubuntu
  • Mark post as helpful
  • send pies
    Number of slices to send:
    Optional 'thank-you' note:
  • Quote
  • Report post to moderator

saeid jamali wrote:Do you think so? are you saying this because of "---" sign? if yea this sign can be any where in the data as you can see tmax, tmin, sunny columns also have this ("---") and I have 4 more files formatted exactly the same but this "---" can be found any where...


OK, so what's wrong with:
?

Winston
 
saeid jamali
Ranch Hand
Posts: 76
  • Mark post as helpful
  • send pies
    Number of slices to send:
    Optional 'thank-you' note:
  • Quote
  • Report post to moderator
EDIT: it doesn't work. it prints the value of the whole line where "---" was found as 0.
 
saeid jamali
Ranch Hand
Posts: 76
  • Mark post as helpful
  • send pies
    Number of slices to send:
    Optional 'thank-you' note:
  • Quote
  • Report post to moderator
I think somehow the problem with "---" is sorted now I'm trying to ignore letters while reading the file. I noticed all of my file have the same length of header and the actual numbers in all of them starts from 8th line. So is there any way to ignore first 7 lines?
 
Marshal
Posts: 8856
637
Mac OS X VI Editor BSD Java
  • Likes 1
  • Mark post as helpful
  • send pies
    Number of slices to send:
    Optional 'thank-you' note:
  • Quote
  • Report post to moderator
On the performance side, probably you don't need Double or Integer objects. Should be enough for you to have primitives. Consider using parseInt and parseDouble instead.
 
Liutauras Vilda
Marshal
Posts: 8856
637
Mac OS X VI Editor BSD Java
  • Likes 1
  • Mark post as helpful
  • send pies
    Number of slices to send:
    Optional 'thank-you' note:
  • Quote
  • Report post to moderator

saeid jamali wrote:is there any way to ignore first 7 lines?

No. But you can read them and do not do nothing with them. For reading you're using readLine method, so think of how to read 7 lines and do not do nothing with them, and only from line 8 start your processing.
 
Consider Paul's rocket mass heater.
reply
    Bookmark Topic Watch Topic
  • New Topic