From what I have read, there was a strong enough association between effeminacy and sexual practice in the public mind to think that there is a suggestive overlap between the two terms.
Pederasty was certainly controversial, but in upper class circles it was broadly tolerated and defended as a social good, as part of a young man’s moral formation, for reasons which I outline in the book. Scholars argue that Greek culture throughout this period was more or less comfortably bisexual. Romans disapproved of pederasty but because it involved freeborn Greek citizens.
You’re right, there are some bad examples of lasting same-sex relationships cited, but you also find in the literature talk of pederastic relationships in particular that endure into later life.
That passage in Philo’s Special Laws is part of a wider condemnation of pagan sexual excess, including a condemnation of pederasty. He perhaps applies the Holiness Code to the flamboyant behaviour associated with temple rituals and festivals, but I don’t think it can be confined to that; and in any case, we can hardly take Philo as an accurate historical interpreter of Torah.