I've heard that physics is really hard and gets harder in year 12. In year 12, you have to do at least Maths 3U (Extension 1) to be able to cope with the work.
As long as you do well and get a good UAI, then it won't really affect your career pathways. For some careers like Medicine, you need...