You are right, it was indeed the streams.
So I changed my upload code to this, and now it works:
// Upload file
InputStream in = new FileInputStream(file);
ftpClient.setFileType(FTPClient.BINARY_FILE_TYPE);
boolean stored = ftpClient.storeFile(fileName ,in);
in.close();
I had ro remove the completePendingCommand() statement to make it work as well.
Thanks