I'm 42 now. Been riding on the street since 19. Never rode "off road" till I was in my late 20's. Had a bike for a few months then everything went during the divorce. Late 30's started riding again. Finaly got to try a few races. It's adictive!!
I found that as far as skill,,,, going somewhere and riding all day on a Saturday or Sunday helped way more than just an hour here and there. I was having to think about things that others (who had ridden non-stop since kids) could do automaticaly. By riding for longer at a time, it helped build muscle memory.
I have now raced both flattrack and some cross country races. I found that in the cross country races, I could often beat faster riders just by riding smart and holding a steady pace. When you ride over your head, you make mistakes that cost time and energy. Back it down just a notch to a steady pace you can hold, ride smooth, focus on not making mistakes. You may do way better than you think.