I can't tell what kind of material that is. Is it flaky like paint or is it plastic or rubbery? It doesn't look like metal. Include year and model bike. Because that drive shaft could be a solid or a sprung shaft, I think. What happens with material if broken in your fingers? I'd at least wash the tube and shaft with something to get all of what ever it is off, out of there.
It looks like paint flakes in the picture. If this is all it is then I'd say somebody painted the swing arm and got a bunch of paint on the inside.