I'm having a really bad issue with some fairly basic UDP communication using java.net.DatagramSocket. Basically, this is what's going on in the application:
Upon startup, open a socket, lets call it 's', on 0.0.0.0 listening to port <someport>
receive a message on s (s.receive), do some processing.
send a message on s (s.send)
The same socket is used for incoming and outgoing traffic -- thus <someport> should be the source port of the UDP packets.
However, sometimes (randomly, but still pretty often), the UDP packets being sent from the application has a source port of 1024, not <someport>, even though we're still using the same socket. This leads to the packet being dropped by some infrastructure or another, and not reaching it's destination.
The problem has been narrowed down to the host where the application is running (tcpdump), and we are having a hard time reproducing this problem outside the production environment (or at least -- the production site).
Our research so far points to issues with the environment, either OS or Java runtime.
We are running java 1.6.0_16 (64 bit) on this:
$ uname -a
Linux 2.6.18-194.32.1.el5 #1 SMP Mon Dec 20 10:52:42 EST 2010 x86_64 x86_64 x86_64 GNU/Linux
$ cat /etc/redhat-release
Red Hat Enterprise Linux Server release 5.5 (Tikanga)
The application is running in a jboss eap 4.3cp08 container.
If anyone have any clues about what may cause this behaviour it will be highly appreciated!