how can i set the layer priority as my action script mc plays over top of my 3d rendered movie

you can't. layers don't exist outside the authoring environment. upon
publication all on-stage objects are assigned a depth (which depends upon the
objects layer) and that determines the object's z-ordering.

you can control the depth of movieclips with actionscript (in as2) using
swapDepths(). in as3, there's a related way to control z-ordering.

I have read the swapDepths function has been replaced by the setChildIndex?
not quite working for me ( and it could be me )

I have developed a star background ( needs to be refined ) but it plays over
top of my plantes which I have added from 3d max
here is the star script : any suggestions on how can I place the stars behind
the planets ?

function Stars() {

for (var i:int = 0; i < 200; i++) {

var newStar:star_mc = new star_mc();
var newStar1:star1_mc = new star1_mc();
var newStar2:star2_mc = new star2_mc();
var randomValue:Number = Math.random()*-.95;
newStar.x = -100+Math.random()*700;
newStar1.x = -100+Math.random()*700;
newStar2.x = -100+Math.random()*700;
newStar.y = -100+Math.random()*400;
newStar1.y = -100+Math.random()*400;
newStar2.y = -100+Math.random()*400;
newStar.scaleX = newStar.scaleY = randomValue;
newStar1.scaleX = newStar1.scaleY = randomValue;
newStar2.scaleX = newStar2.scaleY = randomValue;
newStar.alpha = 1-randomValue;
newStar1.alpha = 1-randomValue;
newStar2.alpha = 1-randomValue;
this.addChild(newStar);
trace(newStar.getDepth());
//setChildIndex (newStar,-1);
this.addChild(newStar1);
this.addChild(newStar2);
}

}
Stars();

yes, as i mentioned, as3 has other ways of dealing with z-ordering. in you
situation, when you add a star to the display list, instead of using
addChild(), use addChildAt() to control the z-ordering of your stars.

now, if your planets are also children of "this", you should be ok. if
planets are not children of "this", then you will need to alter the index of
one of your planet ancestors and one of your start ancestors. those ancestors
are the closest (to stars and planets, resp) that where both ancestors have the
same parent ancestor.