Large records indexing server timeout issue

Hi @irameshdev,

Our Algolia Pro extension doesn’t amend anything in regards to the actual indexing of content, but can change details about what’s indexed like WooCommerce details or whether or not content would be indexed.

That said, getting back on topic. Timeout issues has never really been the easiest thing to handle at times as it does largely depend on the server doing the work.

According to https://www.lifewire.com/error-524-a-timeout-occurred-4782741 a “524 timeout” is one based on Cloudflare integration as well. Are you able to confirm that the site in question is using Cloudflare?

Based on my past support ticket experiences, reducing the amount of records processed at a time hasn’t typically aided, but we can still try if you prefer. I believe we batch things up at increments of 100 at the moment. The filter below would reduce that to 50.

function irameshdev_modify_wpswa_batch_size( $size ) {
    // We default to 100 records at a time. This reduces to 50.
    return 50;
}
add_filter( 'algolia_indexing_batch_size', 'irameshdev_modify_wpswa_batch_size' );

I know that the UI version of batch processing does do what it can to rely on AJAX requests and I believe it makes new requests for every batch, as opposed to one long AJAX request.

I also know that we have some WP-CLI integration if you want to try that, which has much less reliance on say the WP admin being loaded as server resources, and could hopefully allow for more efficient resource usage for the server. However, this is also one that’s typically not aided as well as we’d prefer regarding for sure avoiding timeouts, but it is definitely useful for bash scripting and cron job scheduling that could be done during slower times for site traffic.

More information regarding our WP-CLI integration https://github.com/WebDevStudios/wp-search-with-algolia/wiki/WP-CLI