robots.txt
The file where your site says which crawlers may enter and which folders. Today it also decides whether AI can read you.
Every search engine and every AI sends a program out to read sites. robots.txt is the note on the door saying who gets in.
There is a business decision behind it and it is worth making on purpose: block the AI crawlers and nobody cites you when someone asks ChatGPT who does this in their city. For a local business that is almost always a loss.
Watch one common trap: blocking a page in robots.txt prevents reading it, and if that page carried a tag asking to stay out of Google, that tag cannot be read either.
What to do about it
Type yourdomain.com/robots.txt and check it is not blocking something it should not. If you do not understand the output, ask before touching it.
Other words about the same thing