Interesting post: I obviously meant to come back to this but never did. I must have been traveling at the time, and my hasty answer didn’t really answer the question. My bad. You raise a couple of other points; so let me give a more considered response.
First – when is it tension and when is it compression? Let’s forget any particular pose for now and just look at the concept. (I know you know all this, but for the sake of readers who aren’t familiar with these concepts, we’ll go over them again.)
Tension is the resistance of tissues to being stretched, whereas compression is the resistance felt due to tissues being pressed into each other. By tissues we can include muscles, ligament, bones and joint capsules among several others. Now, each pose is going to have different points of compression and different points of tension. Generally, the muscles that contract to create a pose will be the ones feeling tension when we move in the opposite direction. For example, if we flex our arm, we contract our biceps. When we straighten the arm we stretch our biceps. In this latter position, the biceps will experience tension. In the flexed case, the biceps will experience compression. So, we have a very simple example; now lets look at a much more complicated area, the hips.
Square pose, Lotus and full Pigeon where the front knee is moved outward, all externally rotate the front femur. Any time we bring the foot inward and the knee outward, we are externally rotating the femur in the hip socket. If we use our muscles to create this movement, we will be using our external hip rotators, and there are several of these. And, as we have seen with the bicep example above, we will be stretching the opposite muscles, which are the internal rotators.
The normally talked about external rotators are 6: the piriformis, the obturators internus and externus, the gemellus superior and inferior and the quadratus femoris. There are also a few other muscles that assist with external rotation such as the gluteus maximus and a few of the adductors. These muscles generally shorten when we externally rotate the femur (this gets complicated if we add other movements, such as flexion of the thigh, but I will come back to that later.)
The normal internal rotators of the femur are the gluteus minimus and medius, and they are helped by the adductors brevis, longus and the upper part of the adductor magnus. And, one other interesting muscle assists in the internal rotation of the femur, the tensor fasciae latae. These muscles are stretched when we externally rotate the femur.
Knowing what is stretched and what is compressed by the action of external rotation allows us to pay attention to what stops us. If the internal rotators are long enough to not stop the external rotation, then eventually we will stop externally rotating when we reach compression. So, what will these two stopping reasons feel like? If we are stopped by tension, we will feel it in these internal rotating muscles. Some of us will feel this in the adductors, others like you, will feel this on "the lateral side of the buttock and thigh" (to re-quote Paul Grilley.) These are the glutes and the tensor fasciae latae. So, you are right and I was also correct, but I was not complete in my answer. Everyone is different, so some people will feel the stretch in the adductors while others will feel it on the lateral side of the upper thigh.
[As an aside, and at the risk of going too deep into this discussion, several ligaments that join the femur to the pelvis may also cause tension. Specifically, the 2 key ligaments (iliofemoral and pubofemoral ligaments) are tightened in external rotation of the femur, and if they are short they too can prevent further rotation. This would be felt as a tension deep in the hip socket. Of course, through Yin Yoga we can target those deeper ligaments but I find it challenging to really feel the difference between tension in these ligaments versus compression in the hip socket.]
Okay, earlier I mentioned that this gets complicated when we are flexing the hip too. And all the poses I mentioned generally have a 90-degree flexion (or more if we are doing the sleeping pigeon). When we flex the hips (by bringing the femur closer to the torso) we are adding a stretch to any muscles on the backside of the pelvis. Since many of the external rotators are routed along the back of the pelvis they can feel a stretch even in a flexed position even though they are in external rotation. It is for this reason that the piriformis muscle can be nicely stretched in Pigeon pose, even though Pigeon is an external rotation of the femur. The piriformis is stretched even more if the front leg is adducted while in Pigeon.
Now, what about compression? I think the above does address and describes how you feel what you reported feeling in Swan or Pigeon, but what about folks who have worked through their tension and have reached compression? What will they feel? Often, external rotation is accompanied by abduction of the femur away from the midline of the body. You may see this when someone does Pigeon with the front knee wide out to the side, or someone sitting in Lotus or Square pose with their knees far apart. In these cases, of abduction, we may experience compression where the greater trochanter starts to pinch the flesh between it and the pelvis. (The greater trochanter is the large bony top of the femur that you can feel by placing your hands on the “hip” – and I know that my use of the word “hip” is rather vague, so if you really want to see this visit this picture. http://home.comcast.net/~wnor/xrayslowerlimb.htm
If rotation of the femur does not get stopped by compressing (or pinching) flesh between the femur and the pelvis, eventually the femur (usually its neck) will come into contact with the rim of the hip socket. [Check page 205 of Calais-Germaine’s excellent book, Anatomy of Movement: http://www.amazon.com/Anatomy-Movement- ... 0939616572
Search for these three keywords to find the right page: coxa vara valga] Once again, the sensations of these points of compression may be very difficult to discern because they are so deep in the hip socket, but this is where to look for signs that you have reached your ultimate limit to hip openings, in the hip socket or around that area.
As always, we don’t quite work through all of our tensile resistance when we hit compression. Compression stops us from going deeper in the pose in that direction. We may be able to stretch our tissues more by going around the point of compression, but eventually we reach an end point where we just can’t go any further. We may still feel some tension, but we will also feel a lot of compression too. Trying to keep getting more range of motion once we have reached our ultimate compression point is pointless and can lead to injury. Thus, it is important to ask ourselves, “what is stopping me?” If it is tension, then keep working – you will open up over time. If it is compression, then enjoy the range of motion you achieved but don’t hurt yourself by trying to go deeper.
I hope this is a more complete answer to your question, but now, I do need to finish answering Damolyn’s original question! She asked what she can do to get more open through the hips. Now, if her restriction is coming from tension, and if Square pose isn’t doing it for her, she can try Swan, Sleeping Swan, Shoelace, Eye-of-the-Needle pose or Wall-Eye. They all can work into the same area. Another idea is simply to get off the couch and start living on the floor! One of the best things you can do to open your hips and strengthen your lower back is to sit on the floor as often as you can. Don’t worry about what position to sit in, change it every few minutes, but live on the floor. Eat meals at your coffee table, read while lying in Sphinx, watch TV in Seal, talk on the phone while in Shoelace. If you start to live on the floor, your hips will open up very fast.
If, however, she has reached the point of compression, then her bones are telling her that she is not going to get her knees to the floor in Square pose, and that is perfectly okay! All she has to do is sit up higher, on more cushions, so that she doesn't have to tilt her pelvis or spine to compensate for her tight hips.