Skip to content

Commit 8566327

Browse files
committed
- slug issue fix
- eable robots txt
1 parent 1b0175b commit 8566327

File tree

4 files changed

+4
-51
lines changed

4 files changed

+4
-51
lines changed

content/posts/1. terraform-provider-development-experience.md renamed to content/posts/1-terraform-provider-development-experience.md

File renamed without changes.

content/posts/2. terraform-provider-patching-experience.md renamed to content/posts/2-terraform-provider-patching-experience.md

File renamed without changes.

hugo.toml

Lines changed: 4 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -1,9 +1,13 @@
11
baseURL = 'https://raim.dev/'
22
languageCode = 'en-us'
33
title = 'TIL of Raimbek'
4+
enableRobotsTXT = true
45

56
theme = "til"
67

8+
[permalinks]
9+
posts = '/posts/:title/'
10+
711
[caches]
812
[caches.images]
913
dir = ':cacheDir/images'

themes/til/layouts/robots.txt

Lines changed: 0 additions & 51 deletions
Original file line numberDiff line numberDiff line change
@@ -1,54 +1,3 @@
1-
{{- if (default false site.Params.robotstxt.blockCC) }}
2-
# Block Common Crawl (https://commoncrawl.org/ccbot)
3-
User-agent: CCBot
4-
Disallow: /
5-
6-
{{ end -}}
7-
8-
{{- if (default false site.Params.robotstxt.blockAI) }}
9-
# Block generative AI/LLM bots (source: https://github.com/ai-robots-txt/ai.robots.txt)
10-
User-agent: AI2Bot
11-
User-agent: Ai2Bot-Dolma
12-
User-agent: Amazonbot
13-
User-agent: anthropic-ai
14-
User-agent: Applebot
15-
User-agent: Applebot-Extended
16-
User-agent: Bytespider
17-
User-agent: ChatGPT-User
18-
User-agent: Claude-Web
19-
User-agent: ClaudeBot
20-
User-agent: cohere-ai
21-
User-agent: Diffbot
22-
User-agent: FacebookBot
23-
User-agent: facebookexternalhit
24-
User-agent: FriendlyCrawler
25-
User-agent: Google-Extended
26-
User-agent: GoogleOther
27-
User-agent: GoogleOther-Image
28-
User-agent: GoogleOther-Video
29-
User-agent: GPTBot
30-
User-agent: iaskspider/2.0
31-
User-agent: ICC-Crawler
32-
User-agent: ImagesiftBot
33-
User-agent: img2dataset
34-
User-agent: ISSCyberRiskCrawler
35-
User-agent: Kangaroo Bot
36-
User-agent: Meta-ExternalAgent
37-
User-agent: Meta-ExternalFetcher
38-
User-agent: OAI-SearchBot
39-
User-agent: omgili
40-
User-agent: omgilibot
41-
User-agent: PerplexityBot
42-
User-agent: PetalBot
43-
User-agent: Scrapy
44-
User-agent: Sidetrade indexer bot
45-
User-agent: Timpibot
46-
User-agent: VelenPublicWebCrawler
47-
User-agent: Webzio-Extended
48-
User-agent: YouBot
49-
Disallow: /
50-
{{ end -}}
51-
521
User-agent: *
532
Allow: /
543

0 commit comments

Comments
 (0)