I once spent four days on a single edge.

It was a portrait composite, a woman standing against a stormy sky, her hair blown sideways by wind that existed in neither of the two source photographs. The sky was dramatic. The lighting sold it. But something kept pulling me out of the image every time I looked at it, something around the crown of her head where the hair met the clouds. I kept adjusting the curves, the color grade, the dodge and burn. It wasn’t until I zoomed to 200% that I finally saw it: a faint, clean halo, maybe two pixels wide, tracing her entire silhouette like a watermark. The mask was cutting too hard. The real world doesn’t have edges like that. Nothing in nature is perfectly separated from its background, and when we pretend it is, viewers feel it even if they can’t name it.

That halo is the thing I want to talk about.

The Actual Problem With Most Masks

Most people treat masking as a selection problem. Get the subject selected, invert, done. But selection and masking are not the same task. A selection answers the question: what do I want? A mask answers the question: how does this thing exist in the world?

Hair, fur, fabric, smoke, translucent skin, all of these have edges that are partially transparent, that let some background color through, that blur and feather and change character depending on the light direction. When you mask with a hard brush or rely solely on Select Subject, you’re making a binary decision, pixel in or pixel out, about something that was never binary to begin with.

The math underneath the mask is an opacity value from 0 to 255. Every pixel on your mask layer is telling Photoshop how much of that layer to show. When you paint or auto-generate a mask, you’re assigning those values. The question is whether you’re assigning them with any awareness of what the original photograph was actually doing at that edge.

Where Select Subject Ends and Real Work Begins

Photoshop’s AI selection tools have gotten genuinely impressive. For clean-edged subjects against simple backgrounds, Select Subject with the Object Aware option is often a usable starting point. I said starting point.

After running any automated selection, I immediately open Select and Mask (Alt+Ctrl+R on Windows, Option+Command+R on Mac) and look at the edge with the Overlay view at around 50% opacity. I’m not looking at the overall shape. I’m looking at the quality of the transition. For hair and fine detail, I use the Refine Edge Brush at a radius of around 20 to 40 pixels, depending on the resolution, and I paint only over the complex areas. The brush is essentially asking Photoshop to re-examine that zone and look for the fine structure the initial selection missed.

The output should go to a Layer Mask, not a new layer. Keep your original intact.

From there, I almost never stop. I go into the mask itself (Alt+click on the mask thumbnail to view it directly) and work with a soft brush at 10 to 20% opacity, painting in levels of gray, not black and white, where the edge is complex. A single strand of hair catching light isn’t fully opaque. Painting it with pure white on the mask is a lie. Try painting it at 180 out of 255, which is about 70% white, and see how much more naturally it sits in the composite.

The Channel Method for Hair and Translucency

For difficult hair against busy backgrounds, the automated tools often give up. This is where channel-based masking earns its reputation.

Open your Channels panel and look at the Red, Green, and Blue channels individually. You’re looking for the one with the most contrast between your subject and the background. Duplicate that channel. Then use Curves (Ctrl+M inside the channel) to push the darks down and pull the lights up until your subject reads as close to solid white as possible against a solid black background. Don’t expect perfection; you’ll need to paint in the interior of the subject with a hard white brush and clean up the background with a hard black brush. What you’re preserving with all of this work is that intermediate zone, the transition region where strands thin out and go translucent. When you Ctrl+click the channel thumbnail to load it as a selection and use it as your mask, that transition survives.

This process runs me about 30 to 45 minutes for a complex hair extraction. There is no shortcut that matches it.

When I Learned to Look at the Background First

Six months. That’s how long I spent studying photographs of light on water for an album cover I was hired to do. The artist wanted a figure emerging from a dark lake at night, lit by a single source above, and what I kept getting wrong wasn’t the figure. It was the way the mask had to become semi-transparent at the waterline where the skin and the water surface were physically occupying the same space.

Water doesn’t have a clean edge where a person stands in it. The surface wraps, it reflects the figure above it, and the boundary between body and water is optically complicated. I ended up building a secondary reflection mask, a separate layer running at 40% opacity using Screen blending mode, that simulated the water surface bleeding up onto the lower legs. The mask on that layer was painted entirely by hand, using photo reference I had collected of people in water at night. The reference library I’ve built over fifteen years isn’t decoration. It’s infrastructure.

Luminosity Masks for Light That Actually Behaves

The final step in most of my compositing masks isn’t about the edge of the subject at all. It’s about how light wraps across surfaces that are now in a new environment.

Luminosity masks, generated either manually through channel operations or with a plugin like Lumenzia or TK Actions, let you create masks based on the brightness values of a layer. You can use these to restrict color grade adjustments, dodging, or light wrap effects to only the highlights or midtones of your composite. For light wrap specifically, I run a Gaussian Blur (typically 40 to 80 pixels on a 300 DPI file) on a merged copy of the background, clip it above my subject layer, set it to Screen at around 15 to 25% opacity, and then apply a luminosity mask so the effect reads strongest in the brightest areas where the ambient light would realistically be entering.

The edge of the mask is the edge of your credibility. If that line doesn’t hold, nothing else you do to the image matters.