...using Story Points to measure efficiency of the development organization
This is the first problem: story points are an estimation tool, not a measurement tool. It's a quick and dirty way to get the "gut feel" developers have about the level of effort/complexity involved in getting the story to DONE. That "gut feel" may be close to the mark or it may be way off but it's based on what the developers know about the story when they assign points.
just negated all the work we did to become more efficient
Does the work that you did add value? Did you learn anything from doing that work? Did that work help your team become more efficient? If you answer yes to any of these questions, then how can you feel that you negated that work? You feel you negated work because you are measuring against the wrong thing. You are not using story points for the right purpose.
In a retrospective,
you should go back over the story points originally assigned then see if the developers would change their sizing estimate based on any new knowledge they have gained about the story after working on it during the iteration. This is how you apply learning to future estimation efforts. And you have to remember, these are ESTIMATES; they are NOT commitments. This is a common mistake made by managers who are holding on to the "command and control" mode that is counter to agile principles.
Story points should NOT be used to measure efficiency. Remember Goodhart's Law: "When a measure becomes a target, it ceases to be a good measure." Points are not a goal, they are an indicator.
Advice: Measure Up -- instead of velocity, measure features delivered/accepted per iteration.