Matlab (like Mathematica) is a standalone program for solving math-related problems. It includes a complete programming language to do that. There's probably a Java API for it, too. It's not cheap, though.
It would be easier to help you if you were more specific about what kinds or algorithms you'd like to implement - differentiation, integration, differential equations, ... ?
You might also have a look at Apache Commons Math, an open source library that has a fair number of helpful classes and algorithms.