The thing about space stuff is always the expense (even rich cultures have to expend money or its equivalent on moving mass from one gravity well to another, and if not money then at least energy.)  One of the things that could make such a penal system interesting would be something that would justify sending prisoners out there.  Asteroid mines and such don't make sense because robots are cheaper and you don't have to feed them or keep them alive in order to work.  There could be such places that don't rely on mining.  One possibility is to maybe send criminals somewhere far enough to be "safe" but close enough to justify the expense of sending them there.  Maybe criminals can become involuntary organ donors, but not in the traditional sense...maybe they can be sent somewhere, hooked up to super-advance "galactomatic" technology that keeps them alive while organs are harvested: one criminial, can--for example--be placed in a kind of regeneration vat that's programmed to keep that particular criminal on the kidney assembly line for the length of their prison term...whenever one of his/her kidneys is harvested, he/she begins growing a new one (which gets harvested in turn, and so on and so on and so on).  Just an idea.  Such technology can, after all, be fiddled with in such a way as to cause all kinds of nefarious derring-do; maybe a particular criminal or group of criminals can figure out a way to fiddle with the machinery from inside and thus, escape, and give bounty hunters a reason to get up in the planet-relative morning...