Index Readiness Checker

Check whether a URL has obvious crawl/index blockers before submitting it.