I once spent three hours on a single strand of hair.

Not a clump. Not a section. One strand, backlit, crossing over a sky replacement that needed to feel like it had always been there. The subject was a model photographed against a gray studio backdrop, and I was dropping her into a coastal cliff scene for a book cover. Everything else was holding together. The light was matched. The color grade was dialed. But that one strand of hair was sitting on top of the new sky like it had been cut out with scissors and glued down by a child.

That’s what a bad mask does. It doesn’t announce itself with a glowing outline. It just makes something feel slightly wrong, and the viewer can’t explain why, and they move on without believing the image.

Masking is the discipline that makes or breaks compositing. Not color grading. Not lighting. Not even your source images. The mask is the seam, and if the seam shows, the whole illusion collapses.

Why Photoshop Sees Edges Differently Than Your Eye Does

When you paint a mask manually or let an algorithm generate one, the software is making decisions based on contrast, luminosity, and color difference between the subject and the background. Your eye, on the other hand, uses context, memory, and spatial reasoning. That gap between how a machine reads an edge and how a human perceives it is exactly where most masks fall apart.

The technical term worth knowing here is edge transition. In the real world, almost nothing has a hard edge. Hair scatters light. Fabric has micro-fibers. Even a ceramic vase photographed against a white wall will have a thin zone where the background light wraps around the form and softens the boundary. When you cut a subject out and place it on a new background, you strip away that transition zone, and the brain registers it as wrong even if the conscious mind can’t name the problem.

The fix is not just feathering your mask. Feathering spreads the transparency uniformly, which often looks soft in the wrong way, like someone rubbed petroleum jelly on the edge. What you actually need is a mask that mimics the behavior of light at that specific boundary.

Select and Mask Is a Starting Point, Not a Finish Line

I use Select and Mask in Photoshop as my entry point for most subjects, but I treat its output as a rough draft, not a deliverable. Here is a workflow that has held up across years of movie poster work.

Start by running Select Subject (Adobe Sensei does a reasonable job on well-lit subjects with decent contrast). Then open Select and Mask and switch your view mode to On Black. This is important because it reveals halos and fringe color that On White tends to hide. Set Edge Detection radius to somewhere between 2 and 4 pixels for most subjects. For hair or fur, push it to 8 to 12 pixels and check Smart Radius.

Use the Refine Edge brush at around 20 to 30 percent strength and paint only over transitional areas like flyaway hair, soft fabric, or transparent areas. Do not paint over clean hard edges. That’s where most people go wrong. They paint everywhere and let the algorithm soften things it should leave sharp.

Output the result to a New Layer with Layer Mask. Never merge it. You will be back.

The Luminosity Trick for Problem Edges

For subjects with complex lighting at the boundary, I often pull a luminosity-based mask using Calculations. Go to Image, Calculations, and set both Source channels to the channel with the most contrast between your subject edge and the background. Usually that’s the Blue channel for studio-lit subjects on gray or white. Set Blending to Multiply and Result to New Channel. Load that channel as a selection, invert if needed, and use it to refine your existing mask by painting into it with a soft brush at 40 to 60 percent opacity.

This approach respects the actual light behavior at the edge rather than inventing a boundary where none clearly existed in the original image. It takes more time but it’s the difference between a mask that holds at 100 percent zoom and one that falls apart the moment a client leans in.

The Composite That Still Bothers Me

My first published composite had a shadow going the wrong direction. The light source in the background image was clearly coming from camera left, hard at about a 45-degree angle. But the subject I dropped in had been photographed under overhead softbox lighting with fill, so the shadows underneath were almost nonexistent, and the ones that existed were pointing straight down. I corrected most of it in post but missed one shadow near the subject’s feet. It printed. Nobody mentioned it.

I still look for it when I see that piece. It’s a small thing, and it’s also the reason I now sketch every composite on paper before I open Photoshop. I map the light direction with an arrow. I mark where shadows should fall. I note the quality of light, hard or soft, warm or cool. By the time I’m masking, I already know what the edge should look like because I know where the light is coming from.

A mask isn’t just a cutout. It’s the record of how light behaves at the boundary between two worlds you’re pretending were always one.

Matching Edge Quality to Light Quality

This is the single principle most tutorials skip. The softness of your mask edge should match the hardness of the light source in the destination background. Hard sunlight produces hard-edged shadows and crisp subject boundaries. An overcast sky produces soft shadows and subtle, gradual edges. If you mask a subject sharply and place them in a foggy, overcast scene, the edge will scream. If you place a softly masked subject in direct noon sunlight, they’ll look like they’re dissolving.

Before you refine a single pixel, study the background image. Find a shadow. Look at its edge. That edge quality is your target.

The mask is not a technical task you finish before the real work begins. It is the real work, and every decision inside it should be made with the final image already in your mind.