All of the above will have an effect on the sound, the source, amplification, room acoustics, isolation (slate for speakers, equipment stand), interconnects, power cables, mains supply, etc. Many people do believe in source first to the extent that I've heard of people running a Naim CDS3 (8k CD player) with a Naim Nait 1 (1980s mini-integrated amp) with Linn Kans. Personally I think taking it to that extent is silly, but to a certain level it does ring true.
Once you reach a decent level of amplification and have a good quality source, changing speakers can make a huge difference to the sound. When I first had my Nautilus 803s I was using a Nakamichi AV-10 (integrated surround receiver) with a cheap DVD player (using the Nak's DAC) and moving from B&W CDM7s to 803's completely transformed my system.
That said, I then went on the Naim route, which was initially fairly unfulfilling until I made the jump up to a NAC82 with two HICAP powre supplies and NAP135 monoblock amplifiers which again transformed the system and it really was a high end and enjoyable sound.
I think your A3 and source (?) should be a decent basis for a speaker upgrade, but you'll no doubt catch the bug and want to upgrade the rest... There's certainly no harm in doing it in stages, that's what I find so fun about it... and if you buy second hand you can usually recoup most of your outlay if you buy wisely.
Cheers,
Francis