I'm not an expert on air cooled BMWs, so please forgive me if I'm talking bo11ocks.
However taking the essence of the problem as I understand it -
The starter doesn't willingly engage or disengage sometimes
The starter struggles to turn over the engine
Aren't these both symptoms of having raised the starter gearing by having changed from an 8 tooth pinion to a 9 tooth one? This has effectively raised the gearing by 12.5% so that the starter tries to turn the engine 12.5% faster than it used to, and the engine resists the motor 12.5% more than it used to.
If the teeth on the starter are going past the teeth on the ring gear 12.5% faster then it will be harder for them to find the point of mesh.
Also if the starter gear is effectively 12.5% bigger diameter (on account of being 12.5% bigger circumference) then the backlash is reduced again making it harder for the gears to mesh. This would also account for the teeth on the ring gear looking like they've been hobbed out to a different toothform in the photos.
Finally if the starter is higher geared then it will struggle more to turn the engine over.
I would expect the V shaped notches to be intentional and provide a "lead-in" for the starter pinion teeth as they move into mesh with the ring gear.
Why did you go for a 9 tooth starter pinion?
If I were you I'd go back to a 9 tooth pinion and clean up the burrs with the dremel as you suggest. In the car world it used to be standard practice to shrink the tooth ring onto a flywheel. I'm a bit out of touch with modern cars so not sure if that's still a current practice. However looking at the pictures of both sides of your flywheel there doesn't seem to be a clear line showing where the flywheel stops and the ring gear starts - it looks more like the teeth are machined directly into the flywheel.
Nice machining by the way. If my explanation is wrong I'd be happy to be put right.
Chas - Just seen that you're in Bath. Me too. Oldfield Park.