Inpainting + Masking

Our API also offers the ability to inpaint (the ability to modify a specific section of an image) via masking out a specific area of an image, and then generating new details based on a provided prompt.

In the example below, this is accomplished by providing the gRPC API with a grayscale mask image, where black pixels represent the areas that will be replaced by the prompt, white pixels represent areas that are to be untouched by the inpainting process, and gray pixels result in partial replacement.

Try it out live by clicking the link below to open the notebook in Google Colab!

Python Example

1. Install the Stability SDK package...

bash
pip install stability-sdk

2. Import our dependencies and set up our environment variables and API Key...

python
import os
import io
import warnings

from PIL import Image
from stability_sdk import client
import stability_sdk.interfaces.gooseai.generation.generation_pb2 as generation
from torchvision.transforms import GaussianBlur

# Our host url should not be prepended with "https" nor should it have a trailing slash.
os.environ['STABILITY_HOST'] = 'grpc.stability.ai:443'

# Sign up for an account at the following link to get an API Key.
# https://platform.stability.ai/

# Click on the following link once you have created an account to be taken to your API Key.
# https://platform.stability.ai/account/keys

# Paste your API Key below.

os.environ['STABILITY_KEY'] = 'apikeyhere'

3. Establish our connection to the API...

python
# Set up our connection to the API.
stability_api = client.StabilityInference(
    key=os.environ['STABILITY_KEY'], # API Key reference.
    verbose=True, # Print debug messages.
    engine="stable-diffusion-xl-1024-v1-0", # Set the engine to use for generation.
    # Check out the following link for a list of available engines: https://platform.stability.ai/docs/features/api-parameters#engine
)

4. Import the image we want to work with and a mask to inform which areas of the image we want affected by the diffusion process...

python
img = Image.open('/content/image.png')

mask_i = Image.open('/content/mask.png')

# Feathering the edges of our mask generally helps provide a better result. Alternately, you can feather the mask in a suite like Photoshop or GIMP.

blur = GaussianBlur(11,20)
mask = blur(mask_i)

Imported image:

Imported Mask:

Feathered Mask:

5. Now we can generate a new image, including our mask to affect the specific areas laid out above...

python
answers = stability_api.generate(
    prompt="crayon drawing of rocket ship launching from forest",
    init_image=img,
    mask_image=mask,
    start_schedule=1,
    seed=44332211, # If attempting to transform an image that was previously generated with our API,
                   # initial images benefit from having their own distinct seed rather than using the seed of the original image generation.
    steps=50, # Amount of inference steps performed on image generation. Defaults to 30.
    cfg_scale=8.0, # Influences how strongly your generation is guided to match your prompt.
                   # Setting this value higher increases the strength in which it tries to match your prompt.
                   # Defaults to 7.0 if not specified.
    width=1024, # Generation width, if not included defaults to 512 or 1024 depending on the engine.
    height=1024, # Generation height, if not included defaults to 512 or 1024 depending on the engine.
    sampler=generation.SAMPLER_K_DPMPP_2M # Choose which sampler we want to denoise our generation with.
                                                 # Defaults to k_lms if not specified. Clip Guidance only supports ancestral samplers.
                                                 # (Available Samplers: ddim, plms, k_euler, k_euler_ancestral, k_heun, k_dpm_2, k_dpm_2_ancestral, k_dpmpp_2s_ancestral, k_lms, k_dpmpp_2m, k_dpmpp_sde)
)

# Set up our warning to print to the console if the adult content classifier is tripped.
# If adult content classifier is not tripped, save generated image.
for resp in answers:
    for artifact in resp.artifacts:
        if artifact.finish_reason == generation.FILTER:
            warnings.warn(
                "Your request activated the API's safety filters and could not be processed."
                "Please modify the prompt and try again.")
        if artifact.type == generation.ARTIFACT_IMAGE:
            global img2
            img2 = Image.open(io.BytesIO(artifact.binary))
            img2.save(str(artifact.seed)+ ".png") # Save our completed image with its seed number as the filename.

Resulting image written to <seed>.png:

Note: This is not representative of all of the parameters available for image generation.

Please check out our protobuf reference for a complete list of parameters available for image generation.