We measured the BIST 100: 15 company sites are closed or empty for AI crawlers
None of Borsa Istanbul's 100 largest companies blocks the search crawlers of ChatGPT or Claude in robots.txt. Yet 15 of their sites are effectively closed to AI because of firewalls, bot blocks or empty pages.
What we measured
On 13 October 2026 we measured the websites of all 100 companies in Borsa Istanbul's BIST 100 index, using the company list and web addresses filed with Türkiye's Public Disclosure Platform (KAP). For each site we checked robots.txt rules for AI crawlers, llms.txt, Organization schema on the home page, whether the page reads without JavaScript, and whether it opens for a request carrying an AI crawler's identity. Every site was measured twice: once with a non-browser client, as AI crawlers fetch pages, and once in real Chrome.
Results
- 0 companies block the search and answer crawlers of ChatGPT, Claude or Perplexity in robots.txt.
- 6 block crawlers that collect model training data, most of them only one bot.
- 15 sites are open on paper but closed or empty for crawlers in practice.
- 14 publish an llms.txt file; 28 have no robots.txt at all; 29 of 91 readable home pages carry Organization schema.
Open on paper, closed in practice
Seven sites sit behind a firewall that rejects any non-browser client. Six return 403 to requests identified as GPTBot or ClaudeBot while serving the same page to a browser, most likely a one-click "block AI bots" setting at the hosting provider. Two home pages, Aselsan and ŞOK, are nearly empty without JavaScript, which most AI crawlers do not run.
The missing ID card
llms.txt and Organization schema are the two simple ways a site tells a machine who it is. Most BIST 100 companies use neither. We also found certificate problems on four sites, including one expired certificate.
Limits
The crawler-identity test imitates a bot; we cannot confirm from outside how the real GPTBot is treated. Three sites did not open in Chrome either and are excluded from the rates. Company-level results are in the data file (CSV).
What it means for your business
When someone asks an AI about your company, the answer is built from pages its crawler can read. A security setting that shuts the door, an empty home page or a missing ID card means the answer is built from what others write about you. Check three things: does your site open for a crawler, is the text there without JavaScript, and does the home page carry Organization schema.