Security clearance with marijuana waiver?

I want to try out for the Navy SEALs or Marine Recon after I graduate college in three years. However I told the recruiter that I have smoked marijuana a few times and he said no big deal I would just require a waiver. now i know i could enlist in the navy but would i be able to get the security clearance required to join the SEALs (Secret) with a marijuana waiver? i'm not a pothead. just a guy who tried it a few times.