The reason that a Keq value of 0.53 is incorrect is due to the fact that you can’t do 0.1 - [BrCH2COOH](aq) to calculate [BrCH2COOH](octanol). This is because of the keyword “an AQUEOUS solution of 0.1M bromoacetic acid is shaken with an equal volume of octan-1-ol. This means that the concentration of bromoacetic acid in the entire solution is 0.05 M. Thus, instead of calculating the concentration of bromoacetic acid in water this way, you must instead replace [BrCH2COOH](aq) in the Ka expression with (0.1 - 9.18*10^3 - x) where x is [BrCH2COOH](octanol).
Because the concentration of bromoacetic acid in octanol and in water are not contributing to a singular molarity, it isn’t possible to do a simple 0.1 - [BrCH2COOH](aq) to solve the question.