Crawling site - SEO

Anyone know what the reason for the agree&information page is for as below. Content is identical to the equivalent information page, except there is no header and footer etc. and it gets crawled by SEO, so therefore comes back with errors like no Title, Meta description etc. Is there a way to exclude the crawling of these pages through a line in robots.txt?

Not sure what this page is for
https://ic-express.co.za/index.php?rout ... tion_id=55

And this is the information page
https://www.ic-express.co.za/index.php? ... tion_id=55

The agree page is for the pop up. You can block in robots.txt or see link below.

https://github.com/opencart/opencart/pull/8263

@Addcreative - ahhhh makes sense. Thanks for that, I will block with robots.txt.

AdieB wrote: ↑

Fri Jul 09, 2021 11:33 pm

@Addcreative - ahhhh makes sense. Thanks for that, I will block with robots.txt.

If it's already indexed in a search engine, blocking in robots.txt text won't remove it from the results.

ADD Creative wrote: ↑

Sat Jul 10, 2021 12:09 am

If it's already indexed in a search engine, blocking in robots.txt text won't remove it from the results.

Absolutely right, but will suggest Op to manually request remove the link via Google search console. And then keep the robots.txt update so it wont be index again.

Add this to your robots.txt: