I would first try to find the log4j configuration file and edit it to remove the duplicates.
I think that a general algorithm for printing only unique log entries would be:
The "read next line from log" is a little complicated because you have to read the entire log entry which appears to be multiple physical lines based on what you displayed. The algorithm assumes that duplicate entries will be adjacent. I would be comfortable tackling this in
Java, or perhaps in Python or PHP. Someone might be able to do this in a line or two of lisp. I wouldn't even try this in bash (thought I'm sure it could be done).