Ricky says "don't be a refactoring bigot". And I have to say, his very first point makes no sense to me. He says "don't criticize code", especially if it is already in your project. I say, be a jerk. A big fat jerk. A "your code is a ghetto" style jerk. And this is ESPECIALLY true for code that is already in your project.
Refactoring is itself a kind of criticism: criticism through action. When you say "this code could be more elegant", you are implicitly saying that it is not as elegant as it could be. If you follow this advice to its logical conclusion, you would never refactor the parts of your codebase that need it the most, the parts written by team members under duress, without proper pairing and review, which load up your technical debt.
As I said a couple of weeks ago, if you have something to say, don't be afraid to say it. Growth comes from understanding; you can't help people understand a problem if you ignore it. As a person who has been told several times in the last week that my code could use some, shall we say, "up-elegantizing", I can say without fear of contradiction that that criticism is the only thing that would have convinced me to try and up my game.
So, yeah, don't just refactor every piece of bad code you see just for the sake of doing it. But by all means, when you ARE going to do it, be a jerk about it. Shorten the feedback loop. Don't dance around the issue. Your pair will appreciate the honesty, and your project will be better off as a result.