This is just my personal experience. (3rd year is just what I have heard)
I'll admit that university mathematics is hard, but if you want to do well in anything, you have to put in the effort no matter how hard things can be. Who knows, things may change for you in uni. You might do better, in fact.
Simply doing normal stream courses is hard too, but not as hard as higher stream. In the normal stream, I hear that a lot of questions in the final exam are similar to the ones you do in the tutorial exercises, so it would be doable after all.
I cannot give you a definite answer, but I have tried my best so far. The real answer for you is to actually try it out and see for yourself.
Besides, you don't choose a major stream until the end of first year anyway, so first year is all open for options anyway. You may find out that maths may not be for you and that biology could be a better option.