(Sorry I didn't get this up last night; took longer than I expected )
Winding and Sides
A way to determine what side of a 3D object is currently being viewed is through path winding. Winding represents the direction in which a closed polygon shape is drawn. For winding to be determined, you need at minimum 3 points that, when drawn create a triangle. The winding for such a shape (L, M, N) is the direction in which that shape is drawn as you progress from point L to point M to point N (and effectively back to point L again).
A positively wound shape is a shape that is drawn in a clockwise direction. A negatively wound shape is a shape that is drawn in a counterclockwise direction.
If we place a positively wound triangular shape on a 3D plane and rotate that plane in any way that shows it's reverse side, the winding of that shape from the perspective of the viewer, or as that shape is viewed in 2D space, will be reversed and become negative. This is natural since anything flipped over is the reverse of what it was originally.
The key here, is that this flipping occurs from the perspective of the viewer - as the 3D shape is being viewed in 2D space.
Now, if there was only a way to determine the winding direction of such a path in a 3D plane (as seen in 2D), you would be able to determine what side of that plane is currently visible. Luckily for us, there is a way to do exactly that. In fact, there are a number of ways (one of which, not the best, I describe in the 3D tutorial on kirupa.com). The standard way is to use the cross-product of vectors that make up 2 of a triangular path's sides and check the resulting z value. This is the very same technique used internally in Flash Player for the culling property in the new enhanced Drawing API in Flash Player 10 (Graphics.drawTriangles and GraphicsTrianglePath).
Determining Winding with Cross-product
A cross-product is an operation between two 3D vectors to produce another vector. In this context, vector represents a directional line projected into space - not to be confused with the Vector data type which is a typed Array. The vector resulting from a cross-product is perpendicular to the two original vectors.
The length, or magnitude of the cross-product is proportional to the magnitudes of the original vectors and the angle between them. As the angle gets smaller, so does the magnitude of the cross-product. If/when the vectors cross, the magnitude will be negative.
The angle here is especially important as we apply this towards determining winding. Consider that winding triangle path in a 3D plane. That path is representative of 2 vectors in that plane. At any point where the plane flips to reveal its back side, those vectors, as they are perceived in 2D space, cross creating a negative magnitude.
What this means is that winding direction is inline with the magnitude of the cross-product from the vectors that define the path; when the cross-product magnitude is negative, so is the winding.
With that in mind, we can use three arbitrary points within a plane that define a triangular path, define vectors based on those points as they exist in 2D space from the perspective of the user's view, get their cross-product and check its magnitude to determine winding and in turn what side of the plane we can see.
Specifically, in terms of Flash, this means specifying points in a movie clip that will represent a winding path/two vectors, getting their global 2D positions with localToGlobal, and then performing the cross-product math to determine a resulting magnitude. For the cross-product, the basic applied formula for a cross-product C from vectors A and B (A×B) is:
Remember that a cross-product is between 3D vectors. The vectors we are obtaining from the movie clip are actually 2D based on global locations of the screen. This gives them x and y values, but not a z. For the cross-product then, z for these vectors is 0.
Cx = AyBz - AzBy
Cy = AzBx - AxBz
Cz = AxBy - AyBx
The fact that we're working from source vectors with a 0 z value is important for two reasons. For one, it greatly reduces our operations since in the formulas listed above, we are left with:
Which leaves only z to be calculated. As a result, and this is the second reason, magnitude is equal to the value of z. No extra calculations are required to obtain the magnitude since it lies directly on the axis. This gives us the following formula:
Cx = Ay*0 - 0*By = 0
Cy = 0*Bx - Ax*0 = 0
Cz = AxBy - AyBx
And since winding is negative when magnitude is, we can determine our winding, and hence our side, by checking magnitude > 0.
magnitude = Cz = AxBy - AyBx
So where do vectors A and B come from? We start with points in the 2D coordinate space of the movie clip being transformed in 3D. Those are brought out to global coordinates using localToGlobal which accounts for all transformations 3D and otherwise so we get those points as the user would see them on the screen. Its from these points that the vectors are derived. Vectors, however, are projections from the origin of a 3D coordinate space. For this to work with the points we've obtained from the movie clip, one of the points needs to represent the origin, and the other two the vectors.
So given points L, M, N, assuming L to be our origin, we get:
And of course for z, we have 0. But z isn't important in our calculations so it is being omitted. Plugging this into our magnitude formula we get
Ax = Mx - Lx
Ay = My - Ly
Bx = Nx - Lx
By = Ny - Ly
And that is exactly what was used to determine the frame required to show in the updated makeTwoSided function, just with a little ternary trickery to do it all inline
magnitude = (Mx - Lx)*(Ny - Ly) - (My - Ly)*(Nx - Lx)
By checking whether or not that calculation returns a positive or negative number will let you know whether or not your movie clip is showing its front face or its back face. From that you can change the movie clip to show whatever you want, such as by changing its frame, or even behave in a different manner if you'd like.
var p1:Point = mc.localToGlobal(new Point(0,0));
var p2:Point = mc.localToGlobal(new Point(100,0));
var p3:Point = mc.localToGlobal(new Point(0,100));
var frontFacing:Boolean = ((p2.x-p1.x)*(p3.y-p1.y) - (p2.y-p1.y)*(p3.x-p1.x) > 0);