Evaluating image segmentation models for background removal for Images

Choosing the right segmentation model for background removal

Cloudflare Images is adding background removal via the Transform API, powered by a dichotomous image segmentation model running on Workers AI. The feature isolates the subject of an image from its background, using a soft saliency mask that assigns each pixel a value between 0 and 255 — 0 for fully transparent background, 255 for fully opaque foreground.

Before shipping, we evaluated several open-source segmentation models for both efficiency and accuracy. The goal was to find a model that reliably produces clean subject isolation across a wide range of use cases — from e-commerce product shots against uniform backgrounds to creative tools where users make stickers and cutouts from photos of people and avatars.

Segmentation versus detection

Image segmentation differs fundamentally from object detection. Detection models draw bounding boxes around regions of interest; segmentation models classify every pixel. For background removal specifically, we need a saliency mask that scores each pixel's likelihood of belonging to the foreground, rather than a multi-class label like "dog" or "chair." Multi-class segmentation is better suited for content analysis; binary saliency is what enables clean background extraction.

Four candidate models

We focused on four architectures, each with a different approach to balancing global and local context:

• U²-Net (U Square Net): Trained on the DUST-TR saliency dataset (10,553 images, horizontally flipped to 21,106 training images). It extracts information via a multi-scale approach, analyzing an image at different zoom levels and combining predictions in a single pass. This makes it effective for images with multiple objects of varying sizes.
• IS-Net (Intermediate Supervision Network): From the same authors as U²-Net, it uses a two-step strategy. First, it separates foreground from background, labeling potential objects of interest. Then it refines boundaries for a final pixel-level mask. This initial background suppression produces cleaner edges, especially for complex images with cluttered backgrounds.
• BiRefNet (Bilateral Reference Network): Confirms that small-scale details align with the broader image structure. It starts with a rough salient-object map, refines fine details, then feeds output back to the global context — moving from global to local and back. This yields higher accuracy on high-resolution images, though it requires multiple passes.
• SAM (Segment Anything Model): Meta's extensible model designed for prompt-based segmentation rather than pure saliency detection. It can produce multi-class masks labeling various objects even if they aren't the primary image subject.

U²-Net, IS-Net, and BiRefNet are exclusively saliency detectors. SAM's general-purpose design offers more flexibility but isn't specifically optimized for producing a single foreground/background mask without user prompts.

Scoring segmentation accuracy

To assess model performance, we relied on standard evaluation datasets where human annotators manually trace the ground-truth areas of objects of interest. Each model's predicted mask is compared against this ground truth across three key metrics:

Intersection over Union (IoU)

IoU — also called the Jaccard index — divides the intersection of predicted and ground-truth areas (overlapping foreground pixels) by their union (total pixels in either area). Scores range from 0 to 1; higher values indicate closer overlap. IoU is the most conservative metric, penalizing small boundary mistakes noticeably.

Dice coefficient

Also called the Sørensen–Dice index, this metric gives more weight to shared pixels, even when predicted and actual areas differ in size. The formula doubles the intersection, then divides by the sum of all pixels. Scores also range from 0 to 1, but tend to run higher than IoU for identical predictions because it's more forgiving of size mismatches.

Pixel accuracy

Pixel accuracy simply measures the percentage of pixels correctly labeled as either foreground or background. While intuitive, this metric can be misleading when backgrounds dominate an image.

As an example: an image with 900 background pixels and 100 foreground pixels, where a model correctly identifies only 5 foreground pixels (5% of all foreground), still achieves 90.5% pixel accuracy — yet effectively misses the subject.

Which metric matters most here

IoU is the best choice for applications requiring precise boundaries, such as autonomous driving. The Dice coefficient is preferred when capturing the object matters more than penalizing overshoot, as in medical imaging.

For background removal, we biased toward IoU and Dice over pixel accuracy, since the latter rewards predicting large background regions at the expense of accurately isolating foreground subjects.

Model evaluation: speed and accuracy trade-offs

To compare the candidate models, we ran a series of tests using the open-source rembg library, which exposes all relevant models through a single interface. Each model was asked to output a prediction mask labeling foreground versus background pixels. We pulled test images from two saliency datasets: Humans, which contains over 7,000 images of people with varied skin tones, clothing, and hairstyles, and DIS5K (version 1.5), which spans a broad range of objects and scenes. Where a model had variants pre-trained on specific segmentation tasks (e.g. clothes, humans), we ran tests on both the generalized model and each specialized variant.

Experiments ran on a GPU with 23 GB VRAM to mirror realistic hardware constraints similar to our existing face detection workload. We repeated the tests on a 94 GB VRAM instance as an upper-bound reference for potential speed gains. The larger configuration is typically reserved for more compute-intensive AI workloads, so we treated it as an exploratory comparison rather than a production scenario.

Key trends emerged from the speed measurements. On the smaller GPU, lightweight models were generally faster: U²-Net (176 MB) averaged 307 milliseconds across both datasets, and Is-Net (179 MB) averaged 351 milliseconds. BiRefNet (973 MB) was noticeably slower, averaging 821 milliseconds across its two generalized variants. However, BiRefNet ran 2.4 times faster on the larger GPU, dropping to 351 milliseconds on average — comparable to the smaller models despite its larger size. The lightweight models showed no meaningful speed gain on the larger instance, suggesting that scaling hardware primarily benefits heavier models.

Both datasets revealed a consistent relationship between visual complexity and performance. All models ran faster on the Humans dataset, whose images typically contain a single, relatively uniform subject. The DIS5K dataset includes more objects, cluttered backgrounds, and varying scales, which required more computation for accurate masks. Complexity also hurt accuracy: every model scored higher segmentation accuracy on the Humans dataset.

Specialized variants generally edged out their generalized counterparts in accuracy, but relying on per-input specialization isn't practical for a broad beta service. We therefore favored general-purpose models that perform well without prior classification. This ruled out SAM, which is designed to work with additional prompt inputs; on unprompted segmentation it produced lower accuracy scores and much higher inference times than the other models tested.

All BiRefNet variants were the most accurate. The generalized variants (-general and -dis) matched the accuracy of specialized variants like -portrait. The birefnet-general variant achieved an average IoU of 0.87 and a Dice coefficient of 0.92 across both datasets.

Other models showed more uneven results. The generalized U²-Net model reached an IoU of 0.89 and Dice of 0.94 on Humans, but dropped to 0.39 IoU and 0.52 Dice on DIS5K. The isnet-general-use model performed more consistently, averaging 0.82 IoU and 0.89 Dice across both datasets.

Qualitative testing: edges and subject focus

Quantitative scores only told part of the story. We also observed whether models could interpret both global and local context. In tests with bicycle wheels photographed against interior and exterior backgrounds, lower-scoring models correctly labeled the area around the wheel but failed on the thin spokes, producing masks that included background pixels between them.

_{Photograph by Yomex Owo on Unsplash}

Other tests exposed the opposite failure: clean edges but missed subject detection. Given a photograph of a gray T-shirt on black gym flooring, both generalized U²-Net and Is-Net models isolated only the logo as salient, omitting the rest of the shirt from the mask.

BiRefNet handled both test types well. Its architecture passes information bidirectionally, allowing pixel-level details to inform — and be informed by — the larger scene. This lets it understand how fine edges fit into the broader object. We selected BiRefNet for the beta of background removal based on this combination of accuracy and generalization.

_{Unlike lower scoring models, the BiRefNet model understood that the entire shirt is the true subject of the image.}

Using the segment parameter

Automatic background removal is now available in open beta on the Images API for both hosted and remote images, for all Cloudflare users on Free and Paid plans.

Set the segment parameter when optimizing an image via a specially-formatted Images URL or a worker, and Cloudflare will isolate the image subject, converting the background to transparent pixels. The parameter composes with other optimization operations, as in the URL below:

example.com/cdn-cgi/image/gravity=face,zoom=0.5,segment=foreground,background=white/image.png

That request will:

• Crop the image toward the detected face.
• Isolate the subject, replacing the background with transparent pixels.
• Fill the transparent pixels with solid white (#FFFFFF).

For programmatic workflows, you can bind the Images API to a worker. To demonstrate, we built a simple image editing app for cutouts and overlays, entirely on Images and Workers. It can produce compositions like this one, where background removal isolates a dog and an ice cream cone before overlaying them on a landscape.

_{Photographs by Guy Hurst (landscape), Oskar Gackowski (ice cream), and me (dog)}

The snippet below shows how to overlay images in a worker:

export default {
  async fetch(request,env) {
    const baseURL = "{image-url}";
    const overlayURL = "{image-url}";
    
    // Fetch responses from image URLs
    const [base, overlay] = await Promise.all([fetch(baseURL),fetch(overlayURL)]);

    return (
      await env.IMAGES
        .input(base.body)
        .draw(
          env.IMAGES.input(overlay.body)
            .transform({segment: "foreground"}), // Optimize the overlay image
            {top: 0} // Position the overlay
        )
        .output({format:"image/webp"})
    ).response();
  }
};

Background removal is an iterative step toward letting developers build more interactive products, and we'll keep refining the approach. Full usage details are in the documentation.