Turning down a job offer is no big deal, but since I told contract company B that I'd take the job, I guess I feel a commitment to keep my word.
Are contract companies use to contractors accepting a position and then backing out because they find something better?
I really don't think it would be much of a burden on company B, because they would have a couple of weeks to find someone else before the project started. Also, as I mentioned, it was only a twenty minute phone interview, so they didn't have a lot of time and effort into the screening me.