I think my first choice would be a custom class -- it's up to you whether to make it a RuntimeException subclass or not. Second choice would be IllegalArgumentException. Third choice would be something generic like RuntimeException, and I put throwing OutOfMemoryError last on the list, because the semantics are wrong; normally it's up to the catcher to do something about it, but here there would be no lasting problem.
If I understand correctly, you basically say that the method shouldn't be called with inputData > 4096 bytes? Then I'd throw an IllegalArgumentException - and write my other code so that it doesn't call the method with such inputData. (You could also provide a method isInputDataValid, that returns a boolean instead of throwing an exception and can be called to check wether it would be legal to call the method using the data.)
The soul is dyed the color of its thoughts. Think only on those things that are in line with your principles and can bear the light of day. The content of your character is your choice. Day by day, what you do is who you become. Your integrity is your destiny - it is the light that guides your way. - Heraclitus
subject: Am i entitled to throw an OutOfMemoryError ?