I say let him, but wouldn't do it myself. People do dumb things like turn right without blinkers right into bikes, other cars, pedestrians, ditches, etc.
In this situation I move aside for right turners especially a bike, just being polite I guess. Live and let live. I don't own the road.