Is it a UK or J spec car? Because UKs should have a seperate bigger cooler. If youve gone single then having a bigger cooler is a definate must as the radiator cooler just is not good enough. You should not need to go manual as there has been many people running 500bhp+ on the stock autobox me being one of them. Its easy for the rubber oil lines from the box to the cooler to become brittle and crack which im guessing is what happened, as for the oil being brown this is not good either.
I would reccomend you do the following -
Drain all the fluid
Replace the filter in the transmission with a new one
Top up with fresh fluid
AND GET A COOLER!