You mentioned pant liners but you may still need something to keep the wind off of your legs. If your chest is not well insulated will first notice it that your hands get cold. If you do not keep your legs warm you will first notice it in your feet. Make sure you do everything possible to keep the wind off of your legs. They have a large amount of surface area and it is often neglected by the average rider.
My experience comes from working in a freezer for over 20 years and seeing every new guy have the same problem. Their hands and feet get cold first so the first thing they look at is better gloves and boots. In most cases better gloves and boots may help but they should take a much closer look at what they are doing to keep their chest and legs warmer.