What Is the Screaming Frog and OpenAI Integration?

1. Enable Custom Extraction in Screaming Frog

To allow Screaming Frog to communicate with external APIs like OpenAI, you must first enable Custom Extraction.

How to Access Custom Extraction Settings:

• Open Screaming Frog SEO Spider (paid version required).
• Go to the “Configuration” menu > “Custom” > “Extraction…”

• Choose “Add” to configure a new extraction rule.

• In the dialogue box, select “JavaScript”, which will allow you to run custom logic on the extracted data.

Fields to Prepare for OpenAI Enrichment:

• Image URLs – if you’re generating alt text.
• Page Titles / Meta Descriptions – for rewriting suggestions.
• Visible On-Page Text – to generate summaries or contextual metadata.
  

Make sure each element you want to enrich is extractable via XPath, CSS Path, or JS rendering.

2. Create a Prompt File or Script

You will need a way to communicate your desired task (e.g., “Generate alt text from this image filename”) to the OpenAI API.

Example Prompt to Generate Alt Text:

json

{

  "prompt": "Generate a descriptive alt text for this image based on the filename and page content: [image_filename] - [page_text]",

  "max_tokens": 40,

  "temperature": 0.7,

  "model": "gpt-4"

}

You can save this prompt structure in a .json file or inject it dynamically into Screaming Frog using custom JavaScript extraction or API call templates.

Tips for Better Prompt Engineering:

• Be specific: “Describe what this image likely represents…” works better than “Write alt text.”
• Add context: Include surrounding content or headings to improve accuracy.
• Limit verbosity: Use max_tokens to restrict overly long outputs.
• Use placeholders like [image_filename] and replace them dynamically during the API call.

3. Configure API Access in Screaming Frog

Setting Up the API Call:

Screaming Frog’s Custom JavaScript field allows for direct API calls using fetch or XMLHttpRequest.

Here’s a basic fetch example (can be customized):

javascript

function fetchAltText(imageName, pageText) {

  const prompt = `Generate an SEO-friendly alt text for the image: ${imageName}, with this page context: ${pageText}`;

  

  return fetch('https://api.openai.com/v1/completions', {

    method: 'POST',

    headers: {

      'Authorization': 'Bearer YOUR_OPENAI_API_KEY',

      'Content-Type': 'application/json'

    },

    body: JSON.stringify({

      model: 'gpt-4',

      prompt: prompt,

      max_tokens: 40,

      temperature: 0.7

    })

  }).then(res => res.json())

    .then(data => data.choices[0].text.trim());

}

Replace YOUR_OPENAI_API_KEY with your actual key.

Connecting Screaming Frog to OpenAI’s API:

• In the Custom Extraction interface, paste your script in the JavaScript box.
• Ensure await or promise-handling is supported by the script engine (some logic might need to be pre-processed).
• Screaming Frog will execute this during crawl, but due to API rate limits or async issues, a local proxy script may be needed for bulk automation.

Note: Screaming Frog is not designed for asynchronous JavaScript in large crawls. For bulk tasks, exporting data and using a separate script (e.g., in Node.js or Python) is more reliable.

4. Run the Crawl and Collect Data

How Screaming Frog Uses the Integration:

Once everything is configured:

• Screaming Frog will crawl your site as usual.
• For each relevant field (e.g., images, text blocks), the custom extractor will send a prompt to OpenAI.
• The response (e.g., alt text) will be captured and saved in the custom extraction column.

Monitoring API Calls and Outputs:

• Use Live Log or Custom Extraction Results to verify that data is returning properly.
• Keep an eye on your OpenAI usage dashboard to avoid exceeding your quota.
• You can export a CSV from Screaming Frog to audit results manually.

5. Export and Review Results

Cleaning and QA of Auto-Generated Fields:

• Open the exported CSV or Excel file from Screaming Frog.
• Manually check a sample of rows for quality issues like:
  • Vague or irrelevant alt text
  • Repetitive phrasing
  • Spelling or logic errors
• Optional: use Excel filters or conditional formatting to flag blanks or duplicates.

Adding Results Back into CMS or Site Code:

• For static sites: Copy-paste enriched metadata into your codebase.
• For CMS (like WordPress, Shopify): Use bulk upload tools or scripts to update alt tags, meta descriptions, etc.
• Store any updated fields back into your content management system or headless CMS (if applicable).