File APIs for Java Developers
Manipulate DOC, XLS, PPT, PDF and many others from your application.
http://aspose.com/file-tools
The moose likes Testing and the fly likes How efficiently GenRocket can insert below kind of data. Big Moose Saloon
  Search | Java FAQ | Recent Topics | Flagged Topics | Hot Topics | Zero Replies
Register / Login
JavaRanch » Java Forums » Engineering » Testing
Bookmark "How efficiently GenRocket can insert below kind of data." Watch "How efficiently GenRocket can insert below kind of data." New topic
Author

How efficiently GenRocket can insert below kind of data.

Saurabh Chakrabarty
Greenhorn

Joined: Mar 20, 2014
Posts: 4
We have tables to capture user activities over internet. So we have tables like USERS, ACTIVITIES, ACTIVITY-PARTICIPANTS, MESSAGES, FILES etc.
Our requirement is to insert data for multiple users in database for multiple days.

Let’s assume we need to insert data for 10000 users spread across 6 months.
All users data will be different depending upon some MODEL.
We have different types of Activities (e.g. group chat, voice chat, 1-1 chat etc..)
Also the content/message for each users will vary. They will have different sized messages (with different % of them) for them depending upon some MODEL.
Same is true for FILES also.
Number of participants in activities will be different.
For all of the above the configurations/percentage distribution of data will be passed (as configuration), e.g. no of users, no of days data to populate, no of activities, messages per activities, % activities containing files, percentage of different size & type of files, percentage of different size of messages, percentage of different activity types etc.
For single day data, we may expect 60-100k activities or rows in ACTIVITIES table (ACTIVITY-PARTICIPANTS & MESSAGES will be more depending upon MODEL). And size of data inserted in DB for all tables is approx 3-4 GB per day.

Now my questions are.
1. Will GenRocket be able to insert data like this in DB.
2. What will be the time taken to insert 6 months data (considering total data size of approx 500GB).

Please get back if you need more clarifications.
Saurabh Chakrabarty
Greenhorn

Joined: Mar 20, 2014
Posts: 4
Also, how easy it is to implement DB schema changes for the same set of table with some column level changes or may be few addition/deletion of tables?
Hycel Taylor Iii
GenRocket Founder
Greenhorn

Joined: Feb 24, 2014
Posts: 10
Hi Saurabh,

There's a lot of information here, so I'll do my best to answer your questions one at a time over multiple posts and as I have time to answer them.

This first answer concerns the speed of GenRocket to produce 500GB of data.

We measure how much data GenRocket can generate by rows per second, rows per minute or rows per hour. On a decent computer with a quad processor, GenRocket can generate on average 15,000 rows of data per second, which translates to 900,000 rows per minute or 54,000,000 rows per hour. GenRocket can further produce data on multiple GenRocket runtime instances to increasing the amount of data produced over a shorter period of time (you may like to check out how GenRocket produced 100,000,000 rows of user data in 24 minutes, http://support.genrocket.com/customer/portal/articles/1489429-scalable-test-data-for-big-data).

Thus, if you can calculate or approximate that amount of data each row will produce, you can then approximate how many rows you need to generate and then deduce approximately how much time it will take GenRocket to generate 500GB of test data.
Hycel Taylor Iii
GenRocket Founder
Greenhorn

Joined: Feb 24, 2014
Posts: 10
Hi Saurabh,

This answer concerns reading parameters from a configuration file.

GenRocket can uses the CSVResourceGen or CSVResourceIndexGen to read data as a configuration file to initialize the values of a generator, primarily the RangeGen generator.

Thus, you could create a CSV file containing parameters from which to initialize certain Generator values within of a given Scenario Domain Attribute. The preferred method is to create multiple GenRocket Scenarios with each one having specific Scenario Domain Attribute Generators set to different values.

This also goes to the GenRocket philosophy of having each Scenario concentrate on one specific task of test data generation. You can then execute multiple GenRocket scenarios one after the other to generate all of the data needed. Creating multiple Scenarios decouples and breaks down complex test data solutions into smaller sub tasks making it easier to create and modify the type of test data needed.
Hycel Taylor Iii
GenRocket Founder
Greenhorn

Joined: Feb 24, 2014
Posts: 10
Hi Saurabh,

This answer concerns changing database schema.

If you want to populate data directly into database tables, then you would use GenRocket Domains to model your tables. To do this cleanly, you could import each table's DDL to create its Domain and afterwards add a Generator for each Attribute.

As your database tables change, you can modify, add or delete Attributes within a given Domain. Most often you will change the behavior of a Domain's Attributes in a given Scenario. You can even create new versions of your entire project as you move from one release to another.

GenRocket Domains do not necessarily need to be tied directly to database schema. It all depends on how you choose to use GenRocket generated test data.

For example, generating data that is then used by service methods that contain business logic to populate one or more database tables, greatly decouples GenRocket Domains from a database schema. It also allows the same Domains and Scenarios to be used for load testing, functional testing and integration testing.
 
Consider Paul's rocket mass heater.
 
subject: How efficiently GenRocket can insert below kind of data.
 
Similar Threads
Generating changes in test data
james mail server - cant send email to external accounts
Performance issue while inserting transactions in DB when concurrent users are more
Performance
GenRocket:Is it possible to generate different data in percentage in fractions like 0.45% percent