Post by pipoy » Wed Mar 25, 2020 11:34 pm

Hi,

I just noticed that in the past 2 months, my 2 websites (hosted on the same unmanaged VPS server) are suddenly getting removed in search engines.

Look at the trend.

https://newgearday.com
Image

https://dragon-switch.com
Image

this has been my robots.txt for years.

Code: Select all

User-agent: *
Disallow: /*&filter
Disallow: /*&limit
Disallow: /*&sort
Disallow: /*?route=account/
Disallow: /*?route=affiliate/
Disallow: /*?route=checkout/
Disallow: /*?route=product/search
Sitemap: https://website.com/sitemap.xml
Console also tells me that my sites cannot be crawled. And Sitemaps have errors fetching.
I couldn't force a fetch on google because there is no such button. If I deleted a submitted sitemap, and add new one. The old data will just appear and say Last read was a long time ago.

https://newgearday.com
Image

https://dragon-switch.com
Image


Now I removed the disallow and see if that can help.
New robots.txt

Code: Select all

User-agent: *
Sitemap: https://website.com/sitemap.xml
I also tried Inspect URL, but I get an error that my website cannot be crawled. Force indexing is blocked as well.

Image
Image

I need your opinions about this.
I am not sure if this is an Opencart issue, SEO Module issue or a server issue.

Active Member

Posts

Joined
Fri Mar 04, 2016 12:18 pm

Post by cedcommerceteam » Thu Mar 26, 2020 1:35 pm

pipoy wrote:
Wed Mar 25, 2020 11:34 pm
Hi,

I just noticed that in the past 2 months, my 2 websites (hosted on the same unmanaged VPS server) are suddenly getting removed in search engines.

Look at the trend.

https://newgearday.com
Image

https://dragon-switch.com
Image

this has been my robots.txt for years.

Code: Select all

User-agent: *
Disallow: /*&filter
Disallow: /*&limit
Disallow: /*&sort
Disallow: /*?route=account/
Disallow: /*?route=affiliate/
Disallow: /*?route=checkout/
Disallow: /*?route=product/search
Sitemap: https://website.com/sitemap.xml
Console also tells me that my sites cannot be crawled. And Sitemaps have errors fetching.
I couldn't force a fetch on google because there is no such button. If I deleted a submitted sitemap, and add new one. The old data will just appear and say Last read was a long time ago.

https://newgearday.com
Image

https://dragon-switch.com
Image


Now I removed the disallow and see if that can help.
New robots.txt

Code: Select all

User-agent: *
Sitemap: https://website.com/sitemap.xml
I also tried Inspect URL, but I get an error that my website cannot be crawled. Force indexing is blocked as well.

Image
Image

I need your opinions about this.
I am not sure if this is an Opencart issue, SEO Module issue or a server issue.
Hello mate,

There are several reasons behind site indexing issue -

1) The format you have for robots.txt is not correct, we recommend to make correction

2) We recommend to add <meta name="robots" content="index, follow" /> in head section

We hope it will help you if any concern please let us know.

Thanks!!

Call us at : (+91) -8765210318 , Skype: live:carlmorgan.cedcommerce , Email: support@cedcommerce.com
CedCommerece : Officail Opencart Partners
CedCommerce Official : www.cedcommerce.com
Opencart Services : www.cedcommerce/opencart-services.com
Sell On various Marketplace Integration : Opencart-Extensions
Magenative Mobile App/IOS Development : magenative.com


User avatar
Active Member

Posts

Joined
Wed Aug 09, 2017 9:02 pm

Post by thekrotek » Thu Mar 26, 2020 2:40 pm

Generic question, not related to OpenCart at all.

Professional OpenCart extensions, support and custom work.
Contact me via email or Skype by support@thekrotek.com


User avatar
Expert Member

Posts

Joined
Sun Jul 03, 2016 12:24 am


Post by EvolveWebHosting » Thu Mar 26, 2020 9:58 pm

Just do a Google search for their robots.txt testing tool and then you can login to Webmaster Tools account and find out what the error(s) are with your robots.txt file.

Image


User avatar
Active Member

Posts

Joined
Fri Mar 27, 2015 11:13 pm
Location - Denver, Colorado, USA
Who is online

Users browsing this forum: No registered users and 21 guests