Deleting is very picky. If you tell it to include or exclude specific files, then it deletes only files. If you know which directory contains the *.cache file, you can specify that instead. For example, assume all of the *.cache files are in a directory named temp. Then the delete would look like:
(By the way, your last sentence doesn't match the delete command you gave, so I assumed that the sentence was correct and the delete command contained a typo)
If you are new to Ant and plan to use it for a while, I really recommend you get a book such as Ant in Action. It is really good about explaining all of the intricacies and contains a lot of helpful example projects.