Yep, If you stay in Matala you can avoid the day trippers completely by spending the day on Red Beach, as you say, or on Kommos beach which is on the other side, to the north. Then you can enjoy Matala in the evening.
The ony trouble is that from Plakias to Matala it takes 4 buses, i.e.
Plakias to Bale
Bale to Aghia Galini
Aghia Galini to Phaestos
Phaestos to Matala
Some of it might have to be done by taxi, if the connections don't work.
Greecemad