With every new generation of cards, there will inevitably be a bump to the best available fanless cards (which is as good as it gets in terms of quietness), though it will tend to lag by 6-12 months. Therefore, generally I would recommend looking for the best options available to you while filtering on fanless cooling, or take a look at the active-but-quiet options on a site like QuietPC. When you start to whittle things down, be mindful of the release date of the card and do a little due diligence to see when there might be a new passive/quiet bump coming.
Finally, because I have not been in the market for a quiet GPU for a while, I can't currently make a hands-on recommendation. However, with that said, based on your listed requirements, currently the Sapphire AMD R7 250 Ultimate Fanless would seem to be a good candidate:
The TDP of the R7 250 is listed at 65W which is less than the 5770, so you should be fine in terms of power. Of course, you will need to make sure you have some air flow in your case and enough space for that monster heat sink. How much you need will depend on usage, ambient temperature etc. so it is hard to definitively say you will be OK - sometimes the only way to tell is to try.
As an aside, I would not discount the Nvidia options - they have great Linux support in terms of drivers, and (like CPUs) the GPUs can often have very different thermal profiles. That means that you may be able to get a much better bang for your buck from one manufacturer over the other in terms of quiet operation.