You're kind of mixing metaphors. Computer graphics programs fall into two categories, generally: "paint" programs and "draw" programs. "Paint" programs represent a drawing as a bitmap -- an array of screen pixels. Painting is just a matter of coloring pixels in some color, and erasing is the same as painting, except with white paint.
"Drawing" (or "vector graphics") programs do what you're doing: store figures as sets of points, lines, or other primitives. You can select and move primitives, you can edit them, and you can delete them -- but not "erase" them.
So
you should probably pick one standard metaphor or another, Either represent a drawing as an array of pixels to be colored in, and implement an eraser as a white paintbrush; or use your current representation, but have a "select" tool and wire the "delete" key and "Edit | Clear" menu item up to delete items.