When you have a $300k loan and refinance midway from high to low or low to high interest you will pay the same amount of interest in the end. The reason why low interest rates are bad is because everyone is bidding prices up. Paying $1000k at low interest for the same house is fundamentally worse than paying $300k at high interest.