Site fidelity and philopatry are behavioral adaptations found in many species and their fitness benefits are well documented. The combined population level consequences of site fidelity and philopatry, however, have received little attention despite their importance for understanding spatial patterns in connectivity and population dynamics. We used an integrative approach to explore consequences of fidelity and philopatry on the fine-scale genetic structure of black-tailed deer (Odocoileus hemionus columbianus). We assessed fidelity to seasonal home ranges based on location data from 64 female deer fitted with global positioning system (GPS) collars between 2004 and 2013. We assessed philopatry from mitochondrial DNA (mtDNA) haplotypes using DNA extracted from 48 deer. Results based on location data revealed very small movements and seasonal home ranges together with high site fidelity. Fidelity improved survival; every 1 km increase in mean interlocation distances between consecutive summers increased the risk of mortality by 56.5%. Results from mtDNA sequencing revealed high genetic differentiation (FST > 0.30) and low haplotype sharing among geographic areas separated by as little as 4–10 km. The high genetic differentiation indicated multigenerational periods of philopatric behavior in the matrilineage of black-tailed deer. Combined these results suggest that site fidelity together with strong sex-biased philopatry can create marked short- and long-term demographic isolation and trap matriarchal units as a subset of the larger population with locally determined vital rates. Where such fine-scale population structuring as a consequence of fidelity and philopatry occurs, matrilineal groups might in some cases best serve as the basic units of conservation and management.