Personally, I have enjoyed chem more than physics - not due to the difficulty of either course, but simply because I find the content more interesting (generally). However, if you enjoy physics significantly more than chem, then I'd suggest sticking with physics.