Win a copy of Testing JavaScript Applications this week in the HTML Pages with CSS and JavaScript forum!
  • Post Reply Bookmark Topic Watch Topic
  • New Topic
programming forums Java Mobile Certification Databases Caching Books Engineering Micro Controllers OS Languages Paradigms IDEs Build Tools Frameworks Application Servers Open Source This Site Careers Other all forums
this forum made possible by our volunteer staff, including ...
Marshals:
  • Campbell Ritchie
  • Bear Bibeault
  • Ron McLeod
  • Jeanne Boyarsky
  • Paul Clapham
Sheriffs:
  • Tim Cooke
  • Liutauras Vilda
  • Junilu Lacar
Saloon Keepers:
  • Tim Moores
  • Stephan van Hulst
  • Tim Holloway
  • fred rosenberger
  • salvin francis
Bartenders:
  • Piet Souris
  • Frits Walraven
  • Carey Brown

Map size

 
Ranch Hand
Posts: 348
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
Hi all,

So, my structure is like this :

Map<String, Map<String,Long>> workHour

this represents the workHour of each employee. So each employee will have a map of Map<String, Long> where String represents the date and long is the working hour at that date. So it's like
workHour :
{
"John Doe":{"2015-01-01":6,"2015-01-02":8,...etc}
"Mary":....
"Jane":....
}

each employee will have 365-366 entries (depends on leap year or not). Will this have a huge toll on memory or other aspect if I have, say..1000 employees (365000-366000 entries)
Thanks
 
Rancher
Posts: 989
9
  • Likes 2
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
Why not use an object oriented design and have an Employee class that holds the desired fields? That way your code to manipulate this data will be easy read and maintain. In general, you should first worry about having a good maintainable design before worrying about memory/performance.
 
David Spades
Ranch Hand
Posts: 348
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
ok, but object oriented design also will have roughly the same amount of data, so it will be 1000 Employee objects with 365-366 daily entries each. unless I'm missing something here?
 
Ranch Hand
Posts: 624
9
BSD Java
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
After having a good maintainable design as suggested by E Armitage, you can keep the data in database so there is no need to keep all huge data in RAM at a given point of time.
You need to get only those records to RAM which you will play with.
 
David Spades
Ranch Hand
Posts: 348
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
ok, so this is for reporting and I did populate those object from database.
The aim here is to minimize DB hits without Cache FW for now (other team is still working on that).
So, is there any side effect on this approach?
 
Rancher
Posts: 4625
47
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
If this is for reporting, what exactly are you doing with the data?

That's a lot of data to stick in a single report, for example.
 
David Spades
Ranch Hand
Posts: 348
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
so, each assignment has locations and team members --> employees with their worklogs.
this is for attendance and billable report
so, it is a huge report prepared every mid-year for all the activities of all employees.
 
Saloon Keeper
Posts: 12165
258
  • Likes 1
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
Do you have any good reason to prevent DB hits if the reports are generated twice a year?
 
Tapas Chand
Ranch Hand
Posts: 624
9
BSD Java
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator

David Spades wrote:so, it is a huge report prepared every mid-year for all the activities of all employees.


I hope you are not trying to view the whole report using JSP.In that case JVM will cry(die).
As far as I know such reports are normally generated in Oracle (atleast we use).
JSP is only used to trigger the stored procedure if trigger requires manual intervention.
Data can be exported to some kind of file from Oracle.
 
Marshal
Posts: 69894
278
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
Why are you using a String for the date? Why do you need a date at all? You can use an array for hours worked. I believe you can use long[] as a type in generics, but you can't use long (small L).
Why are you using a Long for hours worked? Unless you expect each of your staff to work > 2³¹ hours daily? Even Ebenezer Scrooge didn't expect Bob Cratchit to work that long.
 
David Spades
Ranch Hand
Posts: 348
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator

Campbell Ritchie wrote:Why are you using a String for the date? Why do you need a date at all? You can use an array for hours worked. I believe you can use long[] as a type in generics, but you can't use long (small L).
Why are you using a Long for hours worked? Unless you expect each of your staff to work > 2³¹ hours daily? Even Ebenezer Scrooge didn't expect Bob Cratchit to work that long.



2 words that all developers understand all too well --> legacy code
it was stored that way since the beginning of time, I suspect from back in 2000 and now, apparently I'm it.
 
Campbell Ritchie
Marshal
Posts: 69894
278
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
And you can't find who decided to use Strings for dates and shoot them?
 
David Spades
Ranch Hand
Posts: 348
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
nope, since they're the ones hiring my team.
they said that using string is enough at that time and easier to manipulate using joda time. used string in anticipation of using other datetime framework.
 
Campbell Ritchie
Marshal
Posts: 69894
278
  • Likes 1
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
It may be possible to copy the entire database to a new database converting the Strings to dates as you go.

Maybe shooting would be too kind. Something more lingering involving boiling oil or honey and fire ants might be more appropriate
 
David Spades
Ranch Hand
Posts: 348
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
problem is, we're given a new box to put our application and the data resides in other box which is not under our control, so....
that's why I've been asking around about ehcache
https://coderanch.com/t/651517/ehcache/caching/ehcache-store-data

I'm only allowed to implement any kind of workaround so long I put it in my box and not changing anything in other boxes. oh well, if it's easy, anybody can do it right? that's why there's always obstacle, be it technical or "non-technical"
 
E Armitage
Rancher
Posts: 989
9
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
Instead of fetching records into your 'box',s memory, rather create a database instance on that box and setup replication so that your box is a slave that gets updated with data (setting this up is trivial for databases like mysql ). Then use database mechanisms to build the report and write it to disk.
 
Greenhorn
Posts: 18
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator

David Spades wrote:problem is, we're given a new box to put our application and the data resides in other box which is not under our control, so....
that's why I've been asking around about ehcache
https://coderanch.com/t/651517/ehcache/caching/ehcache-store-data

I'm only allowed to implement any kind of workaround so long I put it in my box and not changing anything in other boxes. oh well, if it's easy, anybody can do it right? that's why there's always obstacle, be it technical or "non-technical"



What is the DB that you are using?

If its oracle then the data is already cached in the memory in your other box (I don't know about other DB but they should also work the same way).

When returning result set from Oracle DB you just return as ref cursor. Refcursor is pointer to the cursor that need to be executed.

And Java just execute it and gets the data. Java should not be storing the fetched data in a collection, it should be directly sending it to the destination (Report - display or a file etc).

 
David Spades
Ranch Hand
Posts: 348
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
we're using mongoDB for this one and I think creating database is not feasible since tables involved are transactional.
 
Politics n. Poly "many" + ticks "blood sucking insects". Tiny ad:
Building a Better World in your Backyard by Paul Wheaton and Shawn Klassen-Koop
https://coderanch.com/wiki/718759/books/Building-World-Backyard-Paul-Wheaton
    Bookmark Topic Watch Topic
  • New Topic