With part a, it requires a bit of manipulation and converting everything into x terms.
So if we know xy = 400 (area of poster), then y = 400/x
With photograph --> width = x - 10 and length = y - 10 = 400/x - 10
With the dimensions as x - 10 and 400/x - 10, you just multiply and should get the equation given above
With part b, to find the maximum, we need to find the x-value for dA/dx = 0.
Differentiating the equation gives us dA/dx = 4000/(x^2) - 10
Then you solve dA/dx = 0 (also good to prove it is a maximum by finding the 2nd derivative and proving it is negative; negative = concave down = maximum); should be x = 20
Then sub in x = 20 back into the dimensions (x - 10) and (400/x - 10) to find the area, and it should be 10 * 10 = 100cm^2