No rocks and gravel will not hurt your fish, but if your fish have a parasite such as Flukes, your fish will try and get rid of the parasite by rubbing against just about anything to find relief. The parasite would be the issue, not rocks and gravel.