I am currently studing Speer in Modern and i believe that he would have known but one thing that annoys me is using Himmlers speech as an example of how he must have known. He spent two years searching for evidence against the speech and found some document. There is not enougth evidence to say wiether or not he was there. But i agree with the historian Gitta Sereny when in a documentary, she said how there was no way to know, but then he had friends who would've been there and they would've talked about it afterwards.