Caterpillars primarily feed on leaves, which provide them with the nutrients they need to grow and develop.