If 20 people take advantage of your buffer, then you are delayed by a distance of 20 vehicle-lengths + 20 follow-distances. This is about 1000 meters, a distance which you can travel in about 45 seconds. So the net effect of all 20 people merging in front of you is less than a minute delay on your trip. Unless there is an almost constant stream of people merging in front of you, this isn't adding up to more than a percentage point of two of your whole trip.