I don't think open-source by it's nature is incompatible with XP. O-S is about whether the source to a project is made available to its users, and an XP project can distribute the source just as easily as any other methodology. Many O-S projects, however, are very distributed and have very large developer teams, which certainly makes XP harder.
Some of the XP practices are a good fit for this sort of project, though.
Unit testing and continuous integration have already been adopted by many of the successful O-S projects just to prevent them collapsing into chaos. Refactoring seems to be becoming popular too.