Does it matter which Person of the Trinity we pray to?
Yes, I think it matters. But being wrong about doesn't mean that it's in the category of damnable sin and maybe not even in the category of sin at all.
The Holy Spirit is sent into the world, according to John 16, to glorify the Son. And he glorifies the Son by leading us to the Son and causing us to see the Son as the ground for our access to the Father. The Son came to die for our sins in order to bring us to God.
So the pattern that you find almost uniformly—I say almost uniformly—throughout the New Testament is to pray to the Father in the name of the Son by the power of the Holy Spirit. So we're said to pray "in the Spirit" in Ephesians 6:18. "Pray in the Spirit."
"Pray in the name of Jesus," I think means "on the basis of what Jesus has done to make our access to God possible," namely his blood and righteousness. So when I say, "In Jesus' name" at the end of a prayer, I mean "because Jesus died for me and rose again, covered my sins, and imparted and imputed righteousness to me, I have access to the Father." "Because of him"—that's what "In Jesus' name" means.
I know of no example or encouragement to pray... No, no, no. That's not true. I was going to say, "to pray to the Holy Spirit." But "Come Holy Spirit" is not an evil prayer to the Holy Spirit. "Come Holy Spirit."
But if you got into the habit of praying to the Holy Spirit all the time—"You're my Benefactor. I pray to you"—you would be out of sync with the pattern of the New Testament.
So my bottom line answer—and I've been asked this a lot—is to follow in general the pattern of the Bible, namely, pray to the Father in the name of Jesus by the power of the Spirit, that is, in reliance upon the help of the Spirit.
But, from time to time, "Maranatha! Lord Jesus, come!" is not a bad prayer. And "Holy Spirit, fall upon us and grant us a fresh baptism" is not a bad prayer.
So, in general, pray to the Father; but occasionally, to express their Personhood and your own love for them, telling the Spirit and the Son that you love them and that you would like them to come in fullness is a good thing.