You don't have to try to remember it. After a few years of molar calculations I found that I can remember all the atomic numbers and molar masses of the mostly commonly used elements (i.e. left, right, but not middle or bottom).
Although I once had a tutor who remembered EVERYTHING. Like... antimony, atomic number 51, symbol Sb, atomic mass 121.757. Craziness.