In Quebec (Canada), college is an important school transition between high school and university. This study aims to analyze changes in behavior relating to physical activity, fitness, and physical self-perceptions among a cohort of students who took part in a college physical education program. This longitudinal study also verifies associations between these changes and whether they differ with regard to gender. Data came from a cohort of 256 adolescents (girls = 169, boys = 87; 17 ± 3 years old) who took part in 6 rounds of assessment. Results from latent growth curve modeling showed that, during college, students showed positive changes in physical self-worth, perceived cardiovascular endurance, and perceived body image. Moreover, their level of aerobic exercise also increased, with a higher rate of change for girls. Even though boys reported better perceptions regarding the dimensions of physical self-esteem at the beginning of college, the evolution of these perceptions does not differ according to gender. The evolution of perceived physical self-worth is positively associated with perceived cardiovascular endurance. The latter is, in turn, positively associated with the evolution of perceived body image. Finally, the evolution of aerobic fitness shows a positive association with the evolution of perceived cardiovascular endurance and physical self-worth. Despite the decline in physical activity observed among late teens, positive changes in physical self-esteem and fitness were observed among youths who participated in three physical education classes during three semesters in college.