Skip to content

Error uploading data: KeyValue size too large #1525

@hsg77

Description

@hsg77

Error uploading data: KeyValue size too large
How to set hbase. client. keyvalue. maxsize parameter of HBase?

The following code settings don't work either.
HBaseOptions thbaseOptions=new HBaseOptions();
HBaseOperations operations = new HBaseOperations(this.zookeeper,this.tableName,thbaseOptions);
System.out.println("old hbase.client.keyvalue.maxsize="+operations.getConnection().getConfiguration().get("hbase.client.keyvalue.maxsize"));
//display value(old hbase.client.keyvalue.maxsize=10485760) //10M

operations.getConnection().getConfiguration().set("hbase.client.keyvalue.maxsize","1073741824"); //1024M
HBaseDataStore dataStore = new HBaseDataStore(operations,thbaseOptions);
Configuration config=operations.getConnection().getConfiguration();
//config.set("hbase.client.keyvalue.maxsize","1073741824"); //1024M
//dataStore.prepareRecordWriter(config);
System.out.println("new hbase.client.keyvalue.maxsize="+config.get("hbase.client.keyvalue.maxsize"));
//display value(new hbase.client.keyvalue.maxsize=1073741824)
IndexWriter writer = dataStore.createWriter(featureDataAdapter, primaryIndex);

The error message is as follows:
Exception in thread "main" java.lang.IllegalArgumentException: KeyValue size too large at org.apache.hadoop.hbase.client.HTable.validatePut(HTable.java:1545) at org.apache.hadoop.hbase.client.BufferedMutatorImpl.validatePut(BufferedMutatorImpl.java:175) at org.apache.hadoop.hbase.client.BufferedMutatorImpl.mutate(BufferedMutatorImpl.java:146) at mil.nga.giat.geowave.datastore.hbase.operations.HBaseWriter.writeMutations(HBaseWriter.java:94) at mil.nga.giat.geowave.datastore.hbase.operations.HBaseWriter.write(HBaseWriter.java:88) at mil.nga.giat.geowave.datastore.hbase.operations.HBaseWriter.write(HBaseWriter.java:81) at mil.nga.giat.geowave.core.store.base.BaseIndexWriter.write(BaseIndexWriter.java:99) at mil.nga.giat.geowave.core.store.base.BaseIndexWriter.write(BaseIndexWriter.java:72) at mil.nga.giat.geowave.core.store.index.writer.IndexCompositeWriter.write(IndexCompositeWriter.java:49) at com.cwgis.importFeatureHBase.importData(importFeatureHBase.java:120) at com.cwgis.App.import_db(App.java:115) at com.cwgis.App.main(App.java:51)

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions