The impulsive behavior that is often characteristic of adolescence may reflect underlying neurodevelopmental processes. Moreover, impulsivity is a multi-dimensional construct, and it is plausible that distinct brain networks contribute to its different cognitive, clinical and behavioral aspects. As these networks have not yet been described, we identified distinct cortical and subcortical networks underlying successful inhibitions and inhibition failures in a large sample (n = 1,896) of 14-year-old adolescents. Different networks were associated with drug use (n = 1,593) and attention-deficit hyperactivity disorder symptoms (n = 342). Hypofunctioning of a specific orbitofrontal cortical network was associated with likelihood of initiating drug use in early adolescence. Right inferior frontal activity was related to the speed of the inhibition process (n = 826) and use of illegal substances and associated with genetic variation in a norepinephrine transporter gene (n = 819). Our results indicate that both neural endophenotypes and genetic variation give rise to the various manifestations of impulsive behavior.