Post by alexanderaldev » Sun Mar 07, 2021 7:51 pm

Hi guys,
We've got 4 stores with OC 3.0.3.2 and hosted at VPS sever. Like other servers, our server got a limit for iodes and we've wondered how to get some free space by deleting any unused data. All stores are synced with external product feeds and almost half of the store products are automated inserted. That means products added by-product feeds from external dropshipping companies. Half of the products Enabled and Disabled themselves, also deleted automatically.

Overall all stores got more than 30k products. Each sync start at cron job every hour. When one of the cron jobs has disabled and delete products, each Main and additional image of that product remains on the server. Also, different generated thumbs for any product remain to exist at image/cache (based on store options for Image Dimension, it could every 10 thumbs for 1 main or additional image.

We've wondered why there is not any function in OpenCart to set Expire Time for all cache images in OpenCart or to be added some check, that could delete unused cached images delete by themselves after some time.

There is no sense to delete manually the whole cache folder. If it's deleted all images should be generated again and that could take tons of server process time with a really slow down process for the whole store and server.

Any ideas or solutions on how we could fix this misunderstanding?
Thank you in advance to all who wish to participate in the discussion

https://aldev.bg/


User avatar

Posts

Joined
Thu Oct 24, 2019 4:05 pm
Location - Sofia, Bulgaria

Post by mikeinterserv » Sun Mar 07, 2021 9:58 pm

I don't think a time based solution is a good idea
Some products may not be viewed for a long time and many other issues with time based delete
I think the best path is to add code to your product delete routines - there you have the image in product table
When deleting a product you can delete images in cache and catalog using LIKE in the sql query basing the query on the image in the product table.
There is also additional images in product images table you can delete these with product id the same when deleting the product. It may take a little working out but once its done you should have few problems with this method.

Active Member

Posts

Joined
Thu May 28, 2020 6:55 am
Location - Wales

Post by paulfeakins » Mon Mar 08, 2021 7:21 pm

1. What's the inode limit?
2. Do you have any caching extensions installed?
3. Are your sessions filling the /tmp folder?

For quick, professional OpenCart support please email info@antropy.co.uk


User avatar
Guru Member

Posts

Joined
Mon Aug 22, 2011 11:01 pm
Location - Reigate, Surrey, United Kingdom

Post by JNeuhoff » Mon Mar 08, 2021 7:49 pm

When you say that redundant images remain on your server after products are deleted what exactly do you mean by this?
Are you talking about redundant entries in e.g. the DB table 'oc_product_image'? Or is it only about the image/catalog or image/cache folders?

Override Engine * Integrated VQMod * Unused Images Manager * Instant Option Price Calculator * Number Option * Google Rich Snippets * Google Tag Manager * Export/Import Tool * SpamBot Buster * Survey Plus


User avatar
Expert Member

Posts

Joined
Wed Dec 05, 2007 3:38 am


Post by mikeinterserv » Mon Mar 08, 2021 9:01 pm

JNeuhoff wrote:
Mon Mar 08, 2021 7:49 pm
When you say that redundant images remain on your server after products are deleted what exactly do you mean by this?
Well, when you delete a product that's it, it does not delete the image from server why are you confused by this
so exactly what he means is now redundant images remain on server after product is deleted.

Active Member

Posts

Joined
Thu May 28, 2020 6:55 am
Location - Wales

Post by alexanderaldev » Wed Mar 10, 2021 2:07 am

Thanks Mike, I will try again in another way.

It doesn't matter how the products are deleted from your store "all" images remain to exist on the server including Main Images, Additional Images, .tmp thumbs, cached thumbs, and other not used any more images. And that's my question.
I don't see any sense of that. Ok, in some cases could be necessary images of the product (something like an archive) but in most of the cases you delete the product and you may never need the images again. The Image/Cache folder is the other problem. Based on OpenCart logic each Image got X number thumbs which are generated each time a product is viewed in the frontend. For example 10 different sizes thumbs. That means for 5 images for a product we will have 50 thumbs Only in the cache folder. And the only way to delete them is to delete (or rename) the cache folder and the store generates new thumbs again (lol). And if we have 20 000 products? or 100 000? Maybe we will crush our server. Based on how many people are in real-time they will overload process time for "generating" cache images.

In conclusion, I don't see the point in this and it would only lead to problems. Maybe we need to change this logic and include it in the platform update, at least for the photos in the cache folder, or somehow to be able to set time for expiring which will lead to their self-deletion (at least after a certain period).

https://aldev.bg/


User avatar

Posts

Joined
Thu Oct 24, 2019 4:05 pm
Location - Sofia, Bulgaria

Post by pprmkr » Wed Mar 10, 2021 4:31 pm

In admin/model/catalog/product.php after

Code: Select all

public function deleteProduct($product_id) {
Paste :

Code: Select all

		$deleted_images = array();
		
		$first_image_query = $this->db->query("SELECT image from " . DB_PREFIX . "product WHERE product_id = '" . (int)$product_id . "'");
		$product_images = $this->getProductImages($product_id);
		
		if ($first_image_query->row) {
			$product_images[] = array(
				'image' => $first_image_query->row['image']
				);
		}
		
		foreach ($product_images as $image_to_delete) {
			 if (is_file(DIR_IMAGE . $image_to_delete['image'])) {				 
				//unlink(DIR_IMAGE . $image_to_delete['image']);
				$deleted_images[] = basename($image_to_delete['image']);
				$im_part = substr($image_to_delete['image'], 0, strrpos($image_to_delete['image'], '.'));
				$thumbs = glob(DIR_IMAGE . 'cache/' . $im_part. '-*.*');
				foreach ($thumbs as $thumb) {
					if (is_file($thumb)) {
						//unlink($thumb);
						$deleted_images[] = basename($thumb);
					}
				}  
			 }
		}
		
		$this->log->write($deleted_images);
Remove // before unlink to activate that line.
For testing comment out all $this->db->query("DELETE ... and keep log->write, otherwise comment out log->write

Note: If you only want to remove thumbs, only remove // before //unlink($thumb);
Note 2: Make sure that deleted image is not used elsewhere.

User avatar
Active Member

Posts

Joined
Sat Jan 08, 2011 11:05 pm
Location - Netherlands

Post by alexanderaldev » Fri Mar 19, 2021 5:33 pm

pprmkr wrote:
Wed Mar 10, 2021 4:31 pm
In admin/model/catalog/product.php after

Code: Select all

public function deleteProduct($product_id) {
Paste :

Code: Select all

		$deleted_images = array();
		
		$first_image_query = $this->db->query("SELECT image from " . DB_PREFIX . "product WHERE product_id = '" . (int)$product_id . "'");
		$product_images = $this->getProductImages($product_id);
		
		if ($first_image_query->row) {
			$product_images[] = array(
				'image' => $first_image_query->row['image']
				);
		}
		
		foreach ($product_images as $image_to_delete) {
			 if (is_file(DIR_IMAGE . $image_to_delete['image'])) {				 
				//unlink(DIR_IMAGE . $image_to_delete['image']);
				$deleted_images[] = basename($image_to_delete['image']);
				$im_part = substr($image_to_delete['image'], 0, strrpos($image_to_delete['image'], '.'));
				$thumbs = glob(DIR_IMAGE . 'cache/' . $im_part. '-*.*');
				foreach ($thumbs as $thumb) {
					if (is_file($thumb)) {
						//unlink($thumb);
						$deleted_images[] = basename($thumb);
					}
				}  
			 }
		}
		
		$this->log->write($deleted_images);
Remove // before unlink to activate that line.
For testing comment out all $this->db->query("DELETE ... and keep log->write, otherwise comment out log->write

Note: If you only want to remove thumbs, only remove // before //unlink($thumb);
Note 2: Make sure that deleted image is not used elsewhere.
Hi! Great way for deleting images and it works for manually deleted products. Unfortunately we use Universal Import/Export tool which delete products on his own logic and peace of code. I've tested, but obviously Universal doesn't use main model file with public Function for deleteProducts and can't delete the images as it in the default manually way to delete them (from Admin Product Page). Have you got any ideas?

https://aldev.bg/


User avatar

Posts

Joined
Thu Oct 24, 2019 4:05 pm
Location - Sofia, Bulgaria
Who is online

Users browsing this forum: No registered users and 24 guests