Search engine referencing

I recently updated my baamcode blog and expected search engines to pick up the change swiftly, in particular because the www.smplinux.de domain where the blog is stored has been referenced for many years now. To my big surprise, even a week after updating the site, search engines hadn't picked up any changes. So I decided to be more pro-active and help search engines discover the new site content.

IndexNow for instantly indexing new web content in search engines

A web search yielded the IndexNow protocol for precisely that purpose. It is fairly simple to implement as described at the IndexNow Getting Started site:

Generate an API key
Host the API key at the root of the website
Submit the URL(s)
Verify the URL(s)

The latter verification can be performed via Bing webmaster tools or using Google search console.

For the previous step of submitting the URL(s), several alternative approaches are possible. I chose the one based on curl for now, using a simple shell script as follows:

#!/bin/sh

curl -X POST \
  -H "Content-Type: application/json; charset=utf-8" \
  -d '{
        "host": "www.smplinux.de",
        "key": "<api-key>",
        "keyLocation": "https://www.smplinux.de/<api-key>.txt",
        "urlList": [
"https://www.smplinux.de/",
"https://www.smplinux.de/2024/03/04/D4353066-D5B5-4A4F-B20E-1C7BBD43B12A/",
"https://www.smplinux.de/emacs-blogging/",
"https://www.smplinux.de/2024/02/29/E25AF3A7-2317-4BCA-A723-31D75013A0E0/"
        ]
    }' \
  https://api.indexnow.org/IndexNow \
  -w "%{http_code}\n" \
  -o /dev/null

Here is a more detailed tutorial on search engine submission strategies. Further useful tools in the webmaster toolbox comprise a sitemap.xml file or a robots.txt file.