Background: A subgroup of young children with autism spectrum disorders (ASD) have significant language impairments (phonology, grammar, vocabulary), although such impairments are not considered to be core symptoms of and are not unique to ASD. Children with specific language impairment (SLI) display similar impairments in language. Given evidence for phenotypic and possibly etiologic overlap between SLI and ASD, it has been suggested that language-impaired children with ASD (ASD + language impairment, ALI) may be characterized as having both ASD and SLI. However, the extent to which the language phenotypes in SLI and ALI can be viewed as similar or different depends in part upon the age of the individuals studied. The purpose of the current study is to examine differences in memory abilities, specifically those that are key “markers” of heritable SLI, among young school-age children with SLI, ALI, and ALN (ASD + language normal). Methods: In this cross-sectional study, three groups of children between ages 5 and 8 years participated: SLI (n = 18), ALI (n = 22), and ALN (n = 20). A battery of cognitive, language, and ASD assessments was administered as well as a nonword repetition (NWR) test and measures of verbal memory, visual memory, and processing speed. Results: NWR difficulties were more severe in SLI than in ALI, with the largest effect sizes in response to nonwords with the shortest syllable lengths. Among children with ASD, NWR difficulties were not associated with the presence of impairments in multiple ASD domains, as reported previously. Verbal memory difficulties were present in both SLI and ALI groups relative to children with ALN. Performance on measures related to verbal but not visual memory or processing speed were significantly associated with the relative degree of language impairment in children with ASD, supporting the role of verbal memory difficulties in language impairments among early school-age children with ASD. Conclusions: The primary difference between children with SLI and ALI was in NWR performance, particularly in repeating two- and three-syllable nonwords, suggesting that shared difficulties in early language learning found in previous studies do not necessarily reflect the same underlying mechanisms.