Not really.
The hardest or the most abstract thing in Year 11 Physics would probably be learning about vetors and scalar, which I think everyone should be able to manage.
In year 12, yes, it becomes more difficult. But these days, even physics is a literacy-dominated subject, rather than a maths-oriented subject. If the poster is good at English, I reckon he'll do fine.
i see that you seem only to do 2u maths (like me~!! >_<), most of the people who do physics at my school do 3u maths or 4u maths though ...
i personally chose modern history because i have a genuine interest in it (i did do chemistry and 3u maths in year 11 and found the concepts harder to grasp than modern history) and i think your level of enjoyment of a particular subject should ultimately influence your choice~