- search engine crawler
- feed bots
- SEO crawlers
- social crawler
- advertisment checker bot
- archivers
- Googlebot
- Googlebot-Image
- Googlebot-Mobile
- bingbot
- BingPreview
- msnbot
- Yahoo! Slurp
- YandexBot
- YandexImages
- MJ12bot
- Baiduspider
- Feedfetcher-Google
- Feedstripes
- AhrefsBot
- facebookexternalhit
- AdsBot-Google
- ia_archiver
- Wget
- curl
- libwww-perl
- Python-urllib
-
vs_api: needs to save once to create the option -> admin_notice
-
vs_api: option autoload on/off option
-
vs_api: required fields
-
vs_api: settings: return admin_notice
-
vs_api: phpdoc
-
vs_api: i18n
-
vs_api: row, col, size args
-
vs_api: legends! where?
-
vs_api: TODO radios
-
vs_api: TODO multi checkboxes
-
vs_api: HTML textarea + editor (settings-api-tabs-demo-ban megnézni)
-
vs_api: issue: ideas for Tabs (separate pages, 1 page + js hide/show, 1 page + ?tab=)
-
admin notice in case of subdir, parse_url(home URL)
-
At least one "Disallow" field must be present in the robots.txt file. - check for that
- one day transient with fallback to WP records
- file creation instruction: wget -O ABSPATH . "robots.txt" home . "robots.txt"
- subdir installs with path in Site Address (Settings / General)
- no run on Settings / Reading / "Discourage search engines from indexing this site"
- about FIXME: several UA-s and one is "*"
- recommended sitemaps: http://smythies.com/robots.txt http://www.lemgo.net/robots.txt
- video: (you can drag&drop it into the URL field below after emptying that field)
- video: "URL of the remote robots.txt" deafult value is local (more about other defaults)