Peter Brown

+ Follow
since Sep 26, 2008
Merit badge: grant badges
For More
Cows and Likes
Total received
In last 30 days
Total given
Total received
Received in last 30 days
Total given
Given in last 30 days
Forums and Threads
Scavenger Hunt
expand Ranch Hand Scavenger Hunt
expand Greenhorn Scavenger Hunt

Recent posts by Peter Brown

This is as far as I got if anyone needs it, now re-assessing global transactions:

and it works.

I then setup an aop implementation that gets the needed oldurl from the datasource and newurl from a stepExecutionListener. The idea being that I get the first index of the stepExecutionListener array object in the aop proxy - close (if needed) and initialize with new url and the jdbcTemplate is none the wiser...

I then run the Spring Batch job and get the following:

[The 'createCompositeTransaction' and 'Coordinator not recoverable' are two lines I don't see on my test run.]

I have dug around in the Atomikos code and confirmed that Atomikos is removing the named resource on close.
I have tried setting lazy-init="false" with no change.
I have tried altering the properties file url on close thinking the init will re-read the properties file on init - no change.

Kinda want this Switching Datasources dynamically at runtime (I understand boneCP is not XA Transaction compliant)

11 years ago

This is a continuation of an earlier resolved thread - Datasource url database name not known until after a Batch step. I moved over to using the Atomikos connection pool and the url switch is not sticking. I can't get the AtomikosDataSourceBean to refresh or recognize the url change.

The xml config for the datasource bean is:

The class follows the standard:


gives the bean - I make the changes and try

- log shows it initializing and I output the new url to make sure and the first insert for the UymFileJdbcDao inserts in the default url found in the file.

I don't think my issue is the assigning of the url but how to force the AtomikosDataSourceBean to reset or the JtaTransactionManager to refresh?

I have looked at Spring AOP Hot Swappable Target Sources in my wanders but not sure.

Any pointers, again much appreciated.

11 years ago
Okay - I got the Datasource url database name not known until after a Batch step - fixed.

Here is what I have:

In my launch-context.xml (relevant parts)

The 'envTarget' is to allow the job to read different properties files e.g. development / production. I did have to put
a legitimate jdbc url string for the client.udy.url entry as otherwise I got sql errors. (As a precaution I have
created an empty dummy db so nothing returned on reads etc.)

My module-context.xml job section looks like:

VerifyStoreInStepContextTasklet (implements Tasklet) has:

TestBasicDataSource (implements Tasklet) has:

The next step in the job 'udytranStep' is my check to see what records are actually returned - it correctly connects to the batch job assigned db url and returns expected results.
Thanks to Mark for his help and hope this helps another newbie.
11 years ago
Hi Mark - thanks for the help / pointers.

I tried

and on run it is giving me:

so will keep reading / experimenting...

Per the transactionManager - if I am not using the transactional atm concept (i.e. money must be debited at the same time as withdrawn across different databases) but inserts / updates are single database specific I am thinking I can use the 1 transactionManager?
[Within a step might be an insert to a db on another machine but since that step will be tied to a jdbc writer and so its dataSource, commits / rollbacks should be handled by the 1 transactionManager, right?]

So far my ideas are half-baked! ;-)

11 years ago

Been doing my reading (swirling around right now) and can't fix my current issue.

I have 5 datasources, three of which I can fill in 100% through a property file. The other 2 I have everything but the db name at the end of the url.

(An aside: am I correct in reading that I therefore need 5 Transaction Managers? - If so will follow up with a new question thread)

The schema is the same for db1 so no changes to rowmappers etc - just the db name (same principle for db2).

I came across Dynamic DataSource Routing which I worked through and got working as I try and understand how Spring does things.

I got 'USING THE STEP EXECUTION CONTEXT AND PROMOTING THE DATA TO THE JOB EXECUTION CONTEXT' (Chapter 10 Spring Batch in Action) working by creating the db name in
TrackImportFromJobContextTasklet and retrieving it in VerifyStoreInStepContextTasklet.

I tweaked VerifyStoreInStepContextTasklet to see if I could set the udydb datasource url.

In launch-context I have

And the


i.e. The datasource has not been instantiated yet (I think).

The next section in the book 'WRITING IN THE JOB EXECUTION CONTEXT AND READING USING LATE BINDING' won't work as in a different step.

So I am wrong in my thinking about the datasource and I think the code in VerifyStoreInStepContextTasklet breaches the whole separation of concerns?

I like the Dynamic Datasource Routing but can't figure out how to assign the db name which has been determined from an earlier step in the batch.

Any help appreciated. Thanks
11 years ago

Thanks for the reply. (The storing of the file as a blob is for permanent record keeping.)

I'll get on with reading!
11 years ago

I am a newbie to Spring and Spring Batch and don't know enough to know if the following is possible in Spring Batch.... We currently use Talend ETL for this but I am not 100% happy with it.

Database - mySQL. All linux machines.

Clients ftp flat files to multiple ftp servers. Every minute cron query checking scheduler table. On a 'hit' procedure triggered to check each ftp server client's directory for files and start processing. (Question: I could replace the scheduling with Quartz but looks like the cronExpression is set in a xml file so how to centrally manage and add new client processing scheduling params easily i.e. web interface?)

File picked up, backed up on file system and stored as a blob in database table determined by today's date and client Manufacturer (sending data for) abbreviation from scheduler table. e.g. ABC_2012_Nov. So a case of not knowing database until file is picked up. [For future scalability we look up an architecture schema table to determine on which machine this client's dbs are. (Currently all dbs are on one machine but the idea is that should a client's processing demands get too large we can setup another machine, move its dbs over and then update the architecture schema to new machine and away we go...)]

The FTP file processing procedure then, determines from architecture schema which machine to ssh to, does so and executes the next procedure - handing it the row id of logging table. This and subsequent procedures are now running on local client db machine.

Local Procedure looks up logging table row to get client details e.g. client number, file(s) names etc. Each file is then extracted from the ABC_2012_Nov table ready for conversion. To add to this we have instances where instead of one client per file we have a corporate file with multiple clients in the file. Currently this is handled by checking a db table to see if corporate type or not and then file parsed out to be client specific.

So at this point we know each file is client specific and db connection is set up per looped file e.g ABC_14_2012 (14 = client id). File row is converted into POJO and inserted (with duplicate data checks being done) into ABC_14_2012. Some queries to other db tables (table names are dynamic) are run to update/insert lookup row fields. [Though we have a fixed file layout schema not all clients adhere to it exactly. We have some (consistent per file) that are 314 in length, some 324 and some 329 in length. Another group use an edi like file layout - header, detail and footer section so Batch would need to know or determine which ItemReader schema(?) to use to create common to all, file layout POJO object representing file row.]

Next we do a count to see if we have any inserts that need validation and what type of validation e.g. public validation or private validation. If there are any of each - procedure now hands off to both or one etc.

Validation procedure looks up logging table to get client details e.g. which db to connect to and does so dynamically. Then for each row it processes it against the validation rules (examples of rules would be duplicates, invoice date test, quantity test, client test etc., some of which are connecting to another database and querying dynamically named client specific tables for these tests) and provided all pass - insert to other db tables.

Currently each procedure is segmented so that if one step fails - it can be re-run and following steps will then be called. This is why each looks up the logging table based on id given.

What I would like to see happen (if Spring Batch can do it) is:

Each ftp server act as Batch Master (for lack of a better word) that are both checking the same file processing scheduler. On hit each would backup the file(s) if any, inserting the file into the dynamically determined file db. Then hand off process to dynamically determined machine (idea of remote chunking?) while being kept updated via messaging on progress of node processing data (Batch Admin?) which could be seen via web interface.

Other thoughts: some of the db sql lookups ideally would be run in parallel.

As said don't know enough about Spring / Spring Batch to a) know if can be done b) not horrendously complicated forcing very steep learning curve. The 'Use Cases' on the Spring Batch website don't seem close enough to what I want and Spring Batch samples - don't know enough.

Any help / pointers appreciated. Thank you.
11 years ago
Thanks Ernest for explaining.

To complete the post:
I got it to work by replacing lockFile.exists() with true i.e. infinite loop and handling the start and stop via bourne scripts.

The start script puts the java program in the background and records the PID in the proftpd.lock file.
The stop script opens the proftpd.lock file and kills the PID recorded in it.

top monitoring tool is not showing the background java process being an issue so leaving as is....

13 years ago

proFTPD can use a FIFO file as a log. The manual tells me I must start the monitoring fifo script before the proftpd is started. I found in the init.d script the start parameter and inserted:

proftpd start...
touch /opt/ftp/proftpd.lock
java Simple&

[Simple now in background]

Fifo reads:

In the init.d stop script I have:
rm /opt/ftp/proftpd.lock

which I thought would cause Simple to exit as lockFile no longer exists.

Any help - slaps to the side of the head appreciated.
13 years ago
Gregg - you are right of course.

For closure I would appreciate how to pass the gui reference in UrlConnect urlconn = new UrlConnect(); to the second class as I still draw a blank on how to to do this.

Thank you.
14 years ago
Thank you for the very quick reply.

I (understood) needed it as the xml returned in the second class has values that I want to populate certain textfield's text with. Is my thinking convoluted - is there a better way?

Also in the jFrame getJButtonVerify() where it says

what do I put between the ( and )? I have tried Gui thisClass, this without success.

Thank you
14 years ago

I believe I need to instantiate a reference to the JFrame class in the UrlConnect Class - I do not how to do this. My current code gives me an NPE. I have tried to place the reference in the default constructor - no change. Creating a new instance - does not assign the values to the existing JFrame Gui. They are both in the same package. I really need a working example based on the code below. I have read posts and articles and am no clearer in what I need to do. I am using Eclipse VEP. Thank you. [The xml etc works without issue]

Business Class
14 years ago
Now I remember why I didn't try this - got confused...

My updOrder code is

The validateCallback is

If I understand correctly I should be putting the updExtend() inside the "msg == updated" statement. My question is what is the format for passing parameters into validateCallback?

updExtend() has three fields it requires.

Do I do something like this in the updOrder() function?

and if so does my validateCallback look like this:

Which does not seem to make sense to me...

Thanks Gregg.

I will try that as I would like to see it work in Firefox so I can offer both options to end users.