I think I had walls set to 2. Changing it to 1 seems to have fixed the problem.