Official Online Store of the…

Import software technology

One of the requirements of the terms of reference for this project is filling the site with products from an existing Excel spreadsheet file of the “xlsx” type. Let's consider the solution to this problem in more detail.

General view of the initial data

The main part of the initial data is shown in the screenshot.:

The algorithm performs the required data transformation during the import process. For example, unwanted characters are removed (in the screenshot they are displayed as squares with shadows and diamonds).

Ыome of the original data is hidden for clarity.

The problem difficulties description

As you can see, the format of the input file is quite common. Each line contains item data as positional parameters such as description, title, price, SKU, etc. The original images of the products are located on the main website of the customer, in the Prom.ua marketplace. Accordingly, the source file contains the URLs of these images for each product. While importing one product, all its images are downloaded to the server using the cURL package. The resulting internal image code is used to create an image on the target site as a media library item.

The database of these products has a fairly large size – 480 lines. In addition, from 4 to 8 images are specified for each product. As a result, the media library of products includes about two and a half thousand images on the target site. It goes without saying that downloading such a volume of data via the HTTP protocol should not be done in one run of the algorithm, but with the help of iteration technology.

The one iteration algorithm

First, a list of product images is retrieved from the source table. A corresponding cURL request is executed and one media library item is created for each image. A special meta tag is recorded in the database of the target site, which indicates the source URL from which the corresponding image of the newly created element of the media library was obtained. Thus, we prevent repeated downloading of images with subsequent duplication in the media library. The product photo gallery is compiled from the received images. The first image on the list becomes the main image for that product.

After the product images are ready, an element of the online store product database is created. To prevent product duplication, the standard parameter SKU is used. In addition to the main parameters of the product, the meta title and meta description values are placed in the database. The format of this data corresponds to the format of the plugin Yoast SEO.

Split the import into different iterations

A certain chunk of products is imported during one iteration. The size of this chunk is determined dynamically, depending on the iteration time. If the iteration time exceeds the threshold, the iteration is aborted. Such technical solution is necessary in order to exclude an unpredictable result when the execution is forcibly terminated due to exceeding the time limit of the server on which the software is installed.

The import process control

A special settings page is made in the admin panel of the site fot manage the import process:

The above photo clearly shows which control functions were implemented during the development of the import algorithm. The site operator has the ability to manage the source files on the server. Only one file is involved in the import, which was uploaded to the server later than anyone else. The operator can start and stop the import, as well as perform one iteration. Along with the option to iterate from the admin panel of the site, there is a special procedure that can be launched using the CRON scheduler in order to fully automate the process of importing products.