chore: clean up robots.txt by removing outdated meta directives (#1379)

Co-authored-by: Dharamveer singh <[email protected]>
Co-authored-by: ben-weinberg <[email protected]>

Author: 3 people

public/robots.txt

@@ -2,6 +2,3 @@ User-agent: *
 Allow: /
 Sitemap: https://docs.axelar.dev/sitemap-0.xml
 
-
-# Meta directives
-INDEX, FOLLOW, MAX-IMAGE-PREVIEW:LARGE, MAX-SNIPPET:-1, MAX-VIDEO-PREVIEW:-1