Adding to the choir on this. Yes, Gallifrey is very worth it. I will admit that the plots aren’t the best the Whoniverse has produced, but you don’t listen to Gallifrey for the plots as much as for the characters. The journey that Romana II, Leela and Narvin[1] go on across the series is worth the listen. The first three seasons are essentially one long serialized story. This is not a series that can jumped in and out at random. We have a general Gallifrey thread (Gallifrey - The Audio Series), and then each installment of Series 1-3 has its own Audio Club thread. While the twists and turns are well worth the surprise for the first time listeners, I think Series 1 especially gains something on a relisten when you know the twists because you catch more things that a first-time listener would’ve missed. Anyway, enough rambling. The answer to your question is of course: Yes! (Just wait til @sircarolyn sees this. 
)
these really are the main three, especially Leela ↩︎