I don't have the time to do the code to make it work but maybe some general advice will help. This isn't the complete code of the control so it'll probably be hard for anyone to help without knowing the full context of what's going on and spending the time to understand the problem.
I see you're attempting some type of custom control, that's fine I guess but I'm not sure I would try this with so many *Properties for
everything (margins etc.), it gets a little harder to read the code with so many. Tons of math bindings can get quite confusing after chaining them together. Also
you should probably be inheriting/re-using functionality for things like margins from another builtin container type instead of manually dealing with that math yourself.
If you really want to do it as a standard control, check out the code of some of the simpler built in controls that are similar, or open source lib controls such as ControlsFX.
Personally I probably wouldn't even do this as a brand new control unless I knew I was going to be using it all over the place, and wanted very reusable code (or I was releasing one for others to use), because there's a lot that goes into that.
You could achive a similar visual effect by just manipulating shapes like Rectangle inside of say a VBox or a Pane, and simply re-use the already coded JFX logic to deal with things like margins and min/max size etc. By using a parent container class of some kind you can essentially forget about management of things like margins/padding and just inherit that functionality. All you need to do is modify labels/text and a Rectangle shape inside of a regular container of some type, letting the container(s) deal with all that spacing junk. You can also embed sub-containers (VBox/HBox etc.) inside the parent container so numeric labels can be rendered alongside.
Read up a little on the concept of separating your view from model (MVC concepts and so on). I suspect it will make things simpler. Internally there's no reason you need to deal with negatives at all.
You can accept negative or positive values from external code setting the value of this control, and then convert it into a simple 0 to N (N depending on max height) value which is used to determine the height of the rectangle.