I think it's pretty clear from the size of books 3-5 that literal retellings might not fare that well in the theaters.
My kids both liked it, even though some things were clearly very different.
The reinvention of landscape I thought made sense, even though it was jarring at first. The expansive, graded, universally well-lit courtyards of the first two films, for example, make almost everything non-threatening. Even the entrance to Aragog's lair has that unearthly blue light to it that nonetheless suggests everything's going to be ok.
I thought the time-turning business was captured particularly well.
I didn't really care for the reduction of Malfoy to a simpering coward. What I would have wanted, as a director, is to show his increasing frustration at being marginalized by Potter's prominence and exploits. I mean, c'mon, he's absolutely no threat to a boy who's faced Voldemort twice on one level. On another, schoolboy to schoolboy -- Malfoy gets his digs in in Goblet of Fire and Order of the Phoenix, and I would have liked to see hints of that here.
In retrospect, I think Prisoner of Azkaban could have been tedious and routine if it had been done the same way. I think a change in direction was a good idea. That said, I'm still deciding if I like all the liberties taken with the novel.