Add robot.txt file

[amanmahajan7]

https://www.alanwsmith.com/en/37/wa/jz/s1/
https://developers.google.com/crawling/docs/robots-txt/create-robots-txt

Just added the basic file for now

[amanmahajan7]

Move this step to fail early

[nstepien]

I don't think we need one here, there already is one at the root:
https://comcast.github.io/robots.txt

[nstepien]

The robots.txt file must be located at the root of the site host to which it applies. For instance, to control crawling on all URLs below https://www.example.com/, the robots.txt file must be located at https://www.example.com/robots.txt. It cannot be placed in a subdirectory (for example, at https://example.com/pages/robots.txt).

[amanmahajan7]

Ah I see. Completely missed it