A simple question, if we are using a RSA keypair to sign/encoding, and the other side are using DH/DSS also for sign/encoding we'll have some problem to decrypt?
Not necessarily. As long as each side has access to the other's public key and the software you use can accept either type of keys, everything should work.
As for the so-called "garbage" characters at the end, that is often a symptom of incompatible padding schemes. Without more information or source code it is impossible to know.