Age, along with nutrition and genetics, is one of the key factors in producing quality deer. For most management programs, buck deer need to survive until they are at least 5 years old. If young males are continually harvested your deer herd will not have trophy aged bucks.
With this in mind, determining deer age "on the hoof" is not a difficult task. This video outlines physical characteristics of yearling, immature, mature and post mature white-tailed deer. With practice, you can learn how to judge the age of buck deer.
Notice that the antlers ARE NOT a factor in determining age!