I'm pretty sure it won't be that much harder than 4 unit...
I didn't do 4U back in the HSC. However I still managed to get a credit for MATH1151 without studying much and I didn't do 90% of the homeworks lol (I know credit might be a bit low for some of you nerds out there haha)
However you have tons of stuff to learn though. When you do MATH1151 and MATH1251 most of the stuff are pure maths. There is also some probabilities and statistics, as well as some stuffs that you don't think they have anything to do with Actuarial Studies at all... You have tons of homework questions every week and you have twice as much lectures/tutorials than other courses as they are consists of two main topics (algebra and calculus). As long as you work hard or you have talents in Maths then it should be alright.
Just F.Y.I., more than half the people in our lectures don't listen to anything - they either talk, read comics, play computer games... etc. or not come to lecture at all (there are always tons of empty seats after the first few weeks) I guess that's why so many people failed LOL