Contrary to popular belief, the most important thing to consider when biking in the winter is not how to stay warm so much as how to get rid of excess body heat. Staying warm when cycling does not become an issue until the temperature falls below 0 degrees.
Because it is hard to get rid of excessive heat, dressing appropriately is important. The clothes need to be warm enough to stay warm but not so warm that the rider will be drenched in sweat. Sweat can be extremely dangerous in cold weather as it will cause a person to be chilled as soon as the person stops biking.
It is difficult to make general statements of what to wear given the differences in perception of cold. As a result, generalities are much safer.
When dressing to go for a bike ride in cold weather, step outside and gauge your reaction to the weather first. If you step outside and are immediately cold on the torso under your shirt, you do not have warm enough clothing on. Go back in and add a layer.
If you step outside and just feel cool and not cold, you are dressed appropriately. You will warm up as soon as you get going. In this case, it is very important to warm up for the first mile or two before really getting going. Warming up too fast will result in being too warm in a couple of miles. In addition, excessive amounts of sweat will have to be dealt with.
On the initial step outside, if you are warm, you have too many layers of clothing on. Take a layer off and see if that helps. It is not a good idea to start out warm.