Win a copy of Testing JavaScript Applications this week in the HTML Pages with CSS and JavaScript forum!
  • Post Reply Bookmark Topic Watch Topic
  • New Topic
programming forums Java Mobile Certification Databases Caching Books Engineering Micro Controllers OS Languages Paradigms IDEs Build Tools Frameworks Application Servers Open Source This Site Careers Other all forums
this forum made possible by our volunteer staff, including ...
Marshals:
  • Campbell Ritchie
  • Bear Bibeault
  • Ron McLeod
  • Jeanne Boyarsky
  • Paul Clapham
Sheriffs:
  • Tim Cooke
  • Liutauras Vilda
  • Junilu Lacar
Saloon Keepers:
  • Tim Moores
  • Stephan van Hulst
  • Tim Holloway
  • fred rosenberger
  • salvin francis
Bartenders:
  • Piet Souris
  • Frits Walraven
  • Carey Brown

including 169 .jar files

 
Ranch Hand
Posts: 324
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
Hi

I have various java programs which utilize weka mining software out of Waikato University, New Zealand

My programs are compiled and run on Dukascopy's JForex platform

In order to utilize certain weka classifiers, I have to include a handful of .jar files in my programs

I am now experimenting with a new weka package for deeplearning4j - includes the dl4jMlpClassifier

This package is not a maven project and I have been advised that I will have to include all the jars in my project that are available through the release zip files.

When I look at the wekaDeeplearning_v1.5.14 lib folder I see 169 .jar files

How can I easily include such a large number of .jar files into my java program ?

Thank you

Bob M

New Zealand
 
Marshal
Posts: 25682
69
Eclipse IDE Firefox Browser MySQL Database
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
Here you are: Class Path Wild Cards.
 
Bob Matthews
Ranch Hand
Posts: 324
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
Thank's Paul

Bob M
 
Bob Matthews
Ranch Hand
Posts: 324
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
Hi Paul

Have tried but without success................

The following is from Dukascopy's website

Include libraries

In order to use an external library one has to add a @Library annotation to the strategy. There are two options how to refer to the libaries:

   By name, for instance:

 

   in this case the library has to be placed in the strategy files' directory, i.e., IContext.getFilesDir().

   By full path, for instance:

   

in this case one also has to add the @RequiresFullAccess annotation.
************************************************************
I use the latter method which requires FullAccess

when I try something like "C:/.........../wekaDeeplearning4j_v1.5.14/lib/*" and then compile on Dukascopy's platform it says it can't find that path ???

Bob M
 
Paul Clapham
Marshal
Posts: 25682
69
Eclipse IDE Firefox Browser MySQL Database
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator

Bob Matthews wrote:... I have been advised that I will have to include all the jars in my project that are available through the release zip files...



Would it be possible to ask the person who advised you that to clarify what they meant? Seems to me (from what you said there) that they have given you a boatload of libraries to use as strategies, and you then need to choose the specific libraries you're interested in.
 
Bob Matthews
Ranch Hand
Posts: 324
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
Hi Paul

My limited understanding is as follows.....................

Of the many classifiers available in weka, this one - Deeplearning4jMlpClassifier is different in that the code for it is a third party open source

The weka people have created a wrapper for this classifier (whatever that means) but the one main weka.jar file I have used in the past for other classifiers does NOT include the many .jar dependencies for this classifier

And so I have to bring them in myself manually (I think)

Reading the Dukascopy page seems to suggest that I might have to include all 169 .jar files (less those I can identify are for MAC machines, LINUX etc) one by one - a long arduous process

Thus my original question

Bob M
 
Sheriff
Posts: 7108
184
Eclipse IDE Postgres Database VI Editor Chrome Java Ubuntu
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
There seems to be a Maven online repository for DeepLearning4j:

https://mvnrepository.com/artifact/org.deeplearning4j

Can you get the dependencies you need from there?
 
Bob Matthews
Ranch Hand
Posts: 324
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
Thanks Knute

I have not used Maven before........................

The weka people told me that...........This Weka package itself [wekadeeplearning4j] is not a Maven project.
You will have to include all the jars in your project that are available through the release zip files.

So I was a little confused

Bob M

p.s. I have the following out-of-date instructions to follow:-

Using wekaDeeplearning4j in a Maven Project

It is also possible to include this package as maven project.
As of now it is not provided in any maven repository - this is INCORRECT NOW,
therefore you need to install this package to your local .m2 repository:



Now you can add the maven dependency in your pom.xml file



so - question - what instructions should I follow ?

Update: have installed Maven and on reading it seems that I should create a new project (which I have done in Netbeans)
but am now told to add dependencies by updating the pom.xml file - and here I am lost
 
Knute Snortum
Sheriff
Posts: 7108
184
Eclipse IDE Postgres Database VI Editor Chrome Java Ubuntu
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
Bob, be sure to post a new reply; don't edit an old one.  We get no notification that there is something new to look at if you don't.

In Netbeans, find your project, right-click on it, select Open Pom.  Add the following inside of the project tags:
You may have to add <dependencies></dependencies> tags.  For instance, my pom now looks like this:
 
Bob Matthews
Ranch Hand
Posts: 324
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
Thank you Knute for your recent advice

I am slowly getting there

I have opened up a project in Netbeans which was created a year or more ago called JForex-3-SDK and then opened the pom.xml file.

It starts with a comment - it is an example maven pom file which allows the user to build JForex-SDK by using the latest version of dependencies from Dukascopy public repository with command: mvn clean install

Two of the dependencies viz. com.dukascopy.dds2 and com.dukascopy.api have versions which I have updated

I also added the dependency you advised wekaDeeplearning4j version 1.5.14

I then ran "Clean and Build Project"

Output is as follows:-

Failed to execute goal on JForex-3-SDK: Could not resolve dependencies for project com.dukascopy.dds2:JForex-3-SDK.jar:3.6.14: Failure to find nz.ac.waikato.cms.weka.Deeplearning4j:jar:1.5.134 in http://www.dukascopy.com/client/jforexlib/publicrepo/ was cashed in the local repository.....etc.

I can see that it is looking in the dukascopy repository for the Deeplearning4j dependency as that is the only repository in the pom.xml file

I don't know what the trouble is with the dds2 dependency ??

Below is a portion of the rather long pom.xml

 
Bob Matthews
Ranch Hand
Posts: 324
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
I have tried the below code but the 3rd line failed

It is also possible to include this package as maven project.
As of now it is not provided in any maven repository,
therefore you need to install this package to your local .m2 repository:



I understand I need to add a repository for the weka dependency but do not know which repository to add,
and I don't understand how the weka dependency knows to utilize the missing weka repository

Bob M
 
Saloon Keeper
Posts: 6530
160
Android Mac OS X Firefox Browser VI Editor Tomcat Server Safari
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
org.deeplearning4j is part of Maven Central (as witnessed by https://repo1.maven.org/maven2/org/deeplearning4j/); I'm surprised Maven doesn't find it automatically. I'm no expert on Maven (far from it), but try adding the repo like this:

 
Bob Matthews
Ranch Hand
Posts: 324
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
Thanks Tim

I tried your suggestion but same error

Cheers

Bob M
 
Knute Snortum
Sheriff
Posts: 7108
184
Eclipse IDE Postgres Database VI Editor Chrome Java Ubuntu
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator

Bob Matthews wrote:I have tried the below code but the 3rd line failed...


Those commands are for Linux.  You're on Windows, I believe.  Change the last line to:
 
Bob Matthews
Ranch Hand
Posts: 324
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
Hi Knute

yes - When I did try, I tried as you suggested but got many errors

I have gone back to the weka people with some questions

Bob M
 
Bob Matthews
Ranch Hand
Posts: 324
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
Hi Knute and Tim

Thanks for persevering

The person I am talking to at weka is not the author of this package and runs Linux - so no help

When I first ran the 3rd line gradlew build etc, it seemed to perform most of the tasks and then fell over with error messages which I can't reproduce

when I rerun that line I am now getting an IOException error

as usual, I know nothing about using gradle but I do need to get this package onto my machine somewhere, somehow

Bob M

p.s. I am 73 yoa - thus the strange questions from time to time
 
Knute Snortum
Sheriff
Posts: 7108
184
Eclipse IDE Postgres Database VI Editor Chrome Java Ubuntu
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
Have you tried following the instruction here?

https://deeplearning.cms.waikato.ac.nz/install/
 
Bob Matthews
Ranch Hand
Posts: 324
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
yes



is where my build falls over with errors

I have added --debug, --stacktrace. --info options but the output is overwhelming
Am wondering whether there are other prerequisites that I need to do ?

Bob M

p.s. have asked a question on  a Gradle forum also - no replies as yet
 
Bob Matthews
Ranch Hand
Posts: 324
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
error output:-

FAILURE: Build failed with an exception.

*What went wrong:
java.io.IOException: The filename, directory name, or volume label syntax is incorrect

--scan option seems to be the fullest output
 
Bob Matthews
Ranch Hand
Posts: 324
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
Hi Tim

I have just realised that your comments re Maven Central apply to deeplearning4j NOT wekadeeplearning4j which is the package I am trying to access from my java program

Bob M
 
Knute Snortum
Sheriff
Posts: 7108
184
Eclipse IDE Postgres Database VI Editor Chrome Java Ubuntu
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
Did you try following the instructions in the link I gave you?  It involves downloading the Weka jar and Package zip file, then issuing a Java command like this:


where <WEKA-JAR-PATH> must be replaced by the path pointing to the Weka jar file, and <PACKAGE-ZIP> is the wekaDeeplearning4j package zip file.
 
Bob Matthews
Ranch Hand
Posts: 324
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
I used the GUI interface of weka and used the Package Manager tool to download the wekadeeplearning4j package

After that, while still using the GUI interface, I built a Neural Network, trained and tested it and saved it successfully

Now all I wish to do is to to retrieve that model and pass a new Instance to it to classify (outside of the GUI interface) i.e. from my java trading program using weka code

Under normal circumstances, with other weka classifiers [i.e. not 3rd party classifiers], this would involve:-
a) retrieve model
b) deserialize model
c) construct new Instance
d) issue a classify command
e) retrieve the prediction

Bob M
 
Knute Snortum
Sheriff
Posts: 7108
184
Eclipse IDE Postgres Database VI Editor Chrome Java Ubuntu
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator

Bob Matthews wrote:

is where my build falls over with errors


This really isn't a gradle problem as much as it is a Weka problem.  I would still try to contact the Weka support people or a support forum and ask about this.  They will probably want the stack trace output.
 
Knute Snortum
Sheriff
Posts: 7108
184
Eclipse IDE Postgres Database VI Editor Chrome Java Ubuntu
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator

Bob Matthews wrote:Now all I wish to do is to to retrieve that model and pass a new Instance to it to classify (outside of the GUI interface) i.e. from my java trading program using weka code


This also sounds like a Weka support problem, but if you tell me exactly what fails, I might be able to help.
 
Bob Matthews
Ranch Hand
Posts: 324
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
Hi Knute

Comments from weka forum:-

In theory, you should only need the weka.jar as dependency (wekadl4j
package installed, of course) and then load packages and initialize
Weka's GOE:
weka.gui.GenericObjectEditor.determineClasses();

The weka.jar does not contain any wekadl4j or dl4j classes. You need
to add all required jars to your classpath. In case of wekadl4j,
that's gonna be quite a lot...

This Weka package itself [wekaDeeplearning4j] is not a Maven project. You will have to
include all the jars in your project that are available through the
release zip files.

Please note, this only installs the Maven artifacts in your "local"
repository (alongside all other Maven artifacts that you ever
required). This local installation needs to be performed an any
machine that you are wanting to use it on via Maven.


I'm not the author of that package, that's why I said, I'm not
familiar with that package.

> When I run the 3rd line (gradlew build.....) I receive several output errors
> I can't tell you exactly what they were because when I retry the 3rd line I am now getting
>
> Java.io.IOException: The filename, directory name.....is incorrect
>
> Could you confirm that the instructions work on windows 10 platform

I don't use Windows (I'm running Linux Mint on my laptop), so I can't
really confirm.

However, Linux uses forward slashes in paths ("/") and Windows uses
backslashes ("\"). So simply substitute these and try again.
BTW You might need gradle (https://gradle.org/) installed on your
system (not sure whether gradlew.bat automatically downloads it).

Bob M

p.s. my last message to the weka forum - Is there anybody in the weka community who can run the 3 lines of code below and report a successful gradle build  on a Windows 10 machine?



 
Bob Matthews
Ranch Hand
Posts: 324
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
I have posted the following output to the weka forum for comment......................

 
Bob Matthews
Ranch Hand
Posts: 324
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
and second part..................

 
Bob Matthews
Ranch Hand
Posts: 324
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
Have just received this from an individual on weka forum..............

I've tried both on Windows and Linux and can't get it to successfully build
(got past your issue by updating the gradle version but ran into further
issues).

Can you import the wekaDeeplearning4J.jar file into your project instead? It
can be downloaded from the repository releases page
(wekaDeeplearning4j-1.5.14.zip).

hallelujah

Bob M
 
Knute Snortum
Sheriff
Posts: 7108
184
Eclipse IDE Postgres Database VI Editor Chrome Java Ubuntu
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
Did that work?  I hope it did.
 
Bob Matthews
Ranch Hand
Posts: 324
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
No...........

I am back to the 169 .jar files

Bob M
 
Knute Snortum
Sheriff
Posts: 7108
184
Eclipse IDE Postgres Database VI Editor Chrome Java Ubuntu
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
Are you worried about deployment?  Because you could make a fat jar file or use an installer.  

So you have to import those 169 jar files by hand at least once.  After that, what do you want to do with it?
 
Bob Matthews
Ranch Hand
Posts: 324
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
No - not worried at all - this is a private exercise for me only

Not sure how to make a fat .jar file.................

I want to run my  stand-alone java program on the Dukascopy forex platform

Bob M
 
Paul Clapham
Marshal
Posts: 25682
69
Eclipse IDE Firefox Browser MySQL Database
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator

Knute Snortum wrote:So you have to import those 169 jar files by hand at least once.



Which would have taken less than the six days this thread has been running... but maybe some other good has come from the thread.
 
Being a smart alec beats the alternative. This tiny ad knows what I'm talking about:
Building a Better World in your Backyard by Paul Wheaton and Shawn Klassen-Koop
https://coderanch.com/wiki/718759/books/Building-World-Backyard-Paul-Wheaton
    Bookmark Topic Watch Topic
  • New Topic