Be sure that you don't put yourself in a a bad position when pulling off to allow a faster rider past.
You will hear when people are coming up behind you. Everyone is very congenial and let you know by blasting the throttle long before they're right on top of you.
Most folks won't be upset by you taking a few extra yards to find a place to pull off that won't leave you behind a log or tree.
Never put yourself out to appease the person behind you. Any true sportsman will understand your decision.