This week's book giveaway is in the OCAJP 8 forum. We're giving away four copies of OCA Java SE 8 Programmer I Study Guide and have Edward Finegan & Robert Liguori on-line! See this thread for details.
Matlab (like Mathematica) is a standalone program for solving math-related problems. It includes a complete programming language to do that. There's probably a Java API for it, too. It's not cheap, though.
It would be easier to help you if you were more specific about what kinds or algorithms you'd like to implement - differentiation, integration, differential equations, ... ?
You might also have a look at Apache Commons Math, an open source library that has a fair number of helpful classes and algorithms.