I have zero natural aptitude for any sciences or maths, and I found neither chem or physics to be challenging in the sense that they made me sit down and think twice about something the way maths does sometimes. Some concepts make you go "wut" since you're not used to them, but they really aren't very "difficult" conceptually.
If you compare the workload needed for both of them, personally I needed more practice with physics questions since they were more mathematical in nature and I'm horrible at maths. I also found that physics had more content (might be just me sucking that badly at it ;w
, so I'd say physics is harder to study for, based on workload and content.
If you can only choose one out of the two, I'd recommend chemistry. It's so much easier to study for, and scales better too marginally.
Note that these are just personal opinions though from someone who hasn't even finished the preliminary course. I'm just being verbose and detailed because I don't want to start my Othello essay. ;w;