From the oracle docs,
Oracle uses blank characters to fill format elements to a constant width equal to the largest element for the relevant format model in the current session language.
I'm guessing to accomodate the (possible) minus sign oracle pads the 4 digit format model to 5 characters!
So a solution using TO_CHAR is to use a
Format Model Modifier to control this blank padding.
Could also use LPAD -
LPAD definition in Oracle SQL Language Reference lpad(string, length of string returned, character to use to pad).
lpad return a string of either varchar2, nvarchar2, or LOB depending on the datatype of your string.