I think most of us on here have fitted aftermarket head units which bypass the stock amplifier so may not have encountered the problems you're having.
Just like you said it could well be the stock amplifier, obviously I would just be checking the connections at the head unit and amp first. I'm sure there'll be plenty of spare amplifiers on here if you need one.
I've just found a fault finding section in the manual, it might be helpful if you haven't already looked at it.
Body Electrical section (near the bottom)
page 75
http://www.turbosupras.com/pages/pages/technical/jza80/a80tsrm.htm