I'm with Thomas ... as a general rule
you should always store the primary key, not the name. You can view it sort of like OO "encapsulation" where the ORDER simply has a FK (reference) to the PRODUCT, w/out needing to know anything about how the PRODUCT is defined.
In addition to the advantages Thomas pointed out (ease with which product name is updated, ability to have multiple products with the same name), you have a smaller data set. If you later decide that the product name column needs to be lengthened, you affect only the PRODUCT table, and not others.
As for the deletion problem, you probably don't want to ever delete a product. It's much better practice to have a 'status' field in the PRODUCT table; when a product is no longer supported, you can set the 'status' to 'inactive' (or whatever you want to use) so that it no longer appears as an option, but all of the old ORDER records still have a valid reference.