Such an interesting discussion!
There are so many points I would like to comment on, so I will probably miss some.
How do you approach code refactoring in the real world?
Firstly, in the real world, I always treat the team as the only method of delivery, which means that the biggest refactoring effort comes from talking with the team, and deciding on a common level of quality, and then holding each other to it.
That being said, if by "real world" you mean I start with a large chunk of unrefactored legacy system, then I have two answers depending on the context. If you are about to make a change in the legacy code, my answer would be to do refactoring before implementing changes. It comes from the Kent Beck quote:
First make the change easy, then make the easy change
I have a section about that in the book: https://livebook.manning.com/book/five-lines-of-code/chapter-1/v-2/point-8551-54-54-0
If on the other hand, you are just annoyed that the legacy system slows you down, then I discuss this in chapter 9 (which I am working on now). First I isolate the legacy code, then I add monitoring, so I can see which parts are being used a lot, and which not at all. Then, because I like easy decisions, I start with the most and least used parts, and either eliminate, refactor, or rewrite them. Note: this is simplified, but you can look up the detailed version in the book in a month or so.
And everything is iterations in software, the shorter the better.
there is NEVER a perfect code
Actually, I like to say that code can be perfect for a specific change, that is some change is as easy as possible. This means if you know the direction the software is taking, then the code can be perfect for that. Of course, in practice, the direction is not entirely known, and also changes. But based on this principle I tend to spend more time refactoring parts that change often.
Please don't do these, they will be a nightmare, both from all the merge conflicts, but also for the customer who does not see
any value. Another problem with this, following what I write in the book, is that we can only deliver (push to master) once our code is refactored, so putting this in a separate sprint means we cannot put any code on master before this sprint.
there's a chance that inexperienced folks get too carried away and get too "theoretical"
I totally see where this is coming from, however, people grow so much doing this. Going crazy and making some small piece of code (like the guilded rose kata) totally gold plated, refactored over the top is an excellent exercise. Once or twice. Of course, we should not spend all our time doing extreme gold plating, some things will never change once written, and therefore require no refactoring at all. But there is a lot of learning and growth in doing it a few times in the beginning.
Rename, Extract, and Compose Method
These are definitely powerful, but I am missing something that introduces classes in this: Replace type code with classes, introduce strategy pattern, encapsulate data.
his recommendation to wait a while before refactoring
I just want to clarify that I only mean wait until you are ready to push your code to master, which I think you should do every day. This is because while we are writing the code it is often quite fluent which means we can change data structures or approaches easily. I don't want to refactor the code to support some algorithm, if I then change my mind and have to undo the refactoring. Only once I have the desired functionality mostly locked down I start refactoring.
Copy/Paste programming is grounds for immediate termination of employment
I know this is a joke, I just want to go on the record and say: there is a place for c&p while working, just not when you deliver (push to master). It is a super quick way to get something up and running, and refactoring should remove the duplication anyway. Copying something you don't understand should be grounds for termination.
I think you meant "peer reviews" but "pair reviews" works
I love synchronous peer review (ie. pair or mob programming). However, I am not a fan of blocking peer review (ie. pull requests), because they prevent continuous integration, which causes bad merges, and longer lead time. But this is running off a tangent far from refactoring.
the average "experienced guy" in large corporate settings [...] has practically little to no experience in refactoring or unit testing
This is what motivated me to contact Manning.
Let me know how that goes, I am very curious!
capture all potential improvements
Not all improvements are cost beneficial. One of my common pet peeves is when people show me some highly optimized code, which has become completely unreadable in the process. I have to be pragmatic in my job, so I usually say that improvements should be guided by usage statistics. Notice, having a well-refactored code base is not an improvement, it is a necessary condition to keep high velocity, and therefore should be part of developers daily work.
delay refactoring until you're ready to deliver is just not in line with the approach recommended by other experts
If you consider that by deliver I mean the change (ie. push to master), not the product, then I don't agree, the approach I recommend nicely aligns with other experts' approaches, in particular red-green-refactor. I just added a few more steps, like push to master, spike, and work, which were all present although implicit in other approaches.
there may be valid business situations where the business can tell developers to forego refactoring
Like if the software is scheduled to be decommissioned.
then sure, skip refactoring
* delay, not skip.
accumulating debt becomes a liability
Here I take Robert C Martin's opinion: a mess is not technical debt. Reference