When deserializeBinaryBulk, we will call istr.readBig to copy data from source to destination directly without copying to buffer. But if we only deserialize necessary rows, we can not call istr.readBig since the data size is small, so that data will be copied to the buffer first, and then copied to the destination. Therefore, there will be no improvement when only deserialize and copy the non-filtered rows.

tiflash/dbms/src/DataTypes/DataTypeNumberBase.cpp

Lines 231 to 245 in 96ae6d2

    
           template <typename T> 
        
           void DataTypeNumberBase<T>::deserializeBinaryBulk( 
        
               IColumn & column, 
        
               ReadBuffer & istr, 
        
               size_t limit, 
        
               double /*avg_value_size_hint*/) const 
        
           { 
        
               typename ColumnVector<T>::Container & x = typeid_cast<ColumnVector<T> &>(column).getData(); 
        
               size_t initial_size = x.size(); 
        
               x.resize(initial_size + limit); 
        
               size_t size = istr.readBig( 
        
                   reinterpret_cast<char *>(&x[initial_size]), 
        
                   sizeof(typename ColumnVector<T>::value_type) * limit); 
        
               x.resize(initial_size + size / sizeof(typename ColumnVector<T>::value_type)); 
        
           }

	template <typename T>
	void DataTypeNumberBase<T>::deserializeBinaryBulk(
	IColumn & column,
	ReadBuffer & istr,
	size_t limit,
	double /avg_value_size_hint/) const
	{
	typename ColumnVector<T>::Container & x = typeid_cast<ColumnVector<T> &>(column).getData();
	size_t initial_size = x.size();
	x.resize(initial_size + limit);
	size_t size = istr.readBig(
	reinterpret_cast<char *>(&x[initial_size]),
	sizeof(typename ColumnVector<T>::value_type) * limit);
	x.resize(initial_size + size / sizeof(typename ColumnVector<T>::value_type));
	}

Optimize the performance of filtering data #9699

Description

Enhancement

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions