I know how to determine if the mouse enters something like a JButton or JTextField, but is there a way to determine if the mouse enters a certain portion of the JFrame?
You can create a Rectangle(...) and while the mouse is moving in the frame, you use the contains(..) method of the Rectangle class to determine if the mouse just entered "your" portion of the JFrame.
Why create a Rectangle for that? That calculation is easy enough:
For the region with (x1, y1) as the top-left corner and (x2, y2) as the bottom-right corner, and (x, y) the current point.