To write a program that can do a matrix multiplication with multiple threads efficiently, you'll have to look at the algorithm for the matrix multiplication and split it into tasks that can performed independently (i.e. one task should not depend on the outcome of the other task). So have a look at the algorithm and think about how you can split it into tasks that can be performed independently and at the same time.
After you have made a plan, you'll have to implement it. For that you'll have to learn how to program multiple threads in Java. The Java Tutorial has a chapter about concurrency and multi-threading: Lession: Concurrency.