Working memory (WM) suffers from severe capacity limitations that vary greatly both between individuals and across the lifespan. Recent psychophysical work suggests that WM is a finite resource, and that this resource is unevenly distributed across items stored and maintained in WM. Internal representations of sensory stimuli are corrupted by random noise, and this noise is modulated by the amount of resource allocated to a particular item. In turn, the precision of an item in WM is directly related to how much of this resource is being dedicated to it. Currently, the neurobiological basis of this limitation is unknown. Here, we demonstrate that WM accuracy is limited by the size of visual field maps in early visual and frontoparietal cortex. First, we simulated population activity in a bump attractor network using various population sizes to examine the relationship between the size of the population and the fidelity of the WM representation. Results show that as population size increases, the peak amplitude of the WM representation increases while the variance of the WM representation decreases. This is consistent with our hypothesis that population size may underlie limitations in WM precision. Second, we used nonlinear population receptive-field mapping to identify visual field maps in early visual cortex as well as parietal and frontal cortices in both hemispheres for each subject. This allowed us to quantify the size of specific populations in individual subjects. We then correlated the size of each visual field map with individual performance in the contralateral visual field during a spatial WM task and found that the size of multiple visual field maps predicted WM precision. Finally, we simulated population level activity in each visual field map within each subject. This simulation confirms the predictions of our theoretical model: the variance of the WM representation in the population activity correlates with WM accuracy. These results strongly support the hypothesis that WM resources are limited by the size of visual field maps, specifically in brain regions known to be critical for WM performance. Our results provide key insights into the neurobiological basis of resource limitations in WM and suggest that such limitations have a structural basis.