QA testing of ML functionality in an upgrade from 6.7.1 to 7.0.0-rc2 revealed an issue where moving shards from 6.7.1 to 7.0.0-rc2 nodes were getting stuck in the translog phase. The reason for this was that the 7.0.0-rc2 would not be able to parse the documents from the 6.7.1 node, internally throwing a MapperParsingException of the form failed to parse field [@timestamp] of type [date] in document with id 'AVr3KgpQqy1yzebk-FVk'. This would in turn make the peer recovery hang, as the node with the MapperParsingException would indefinitely wait for a cluster state update with an updated mapping. This issue does not only present a problem for rolling upgrades, but also does not allow any peer recovery of that index in a pure 7.x cluster unless the translog with the corresponding documents is completely purged.
The problematic dateformat field in the mappings of the index looked as follows:
"@timestamp" : {
"type" : "date",
"format" : "yyyy-MM-dd'T'HH:mm:ss.SSSZ"
}, ...
The documents were of the form
{"@timestamp":"2013-01-25T20:16:55.000Z", ...}
The problem seems to be an incompatibility between 6.x and 7.x when it comes to date format parsing, possibly related to #27330
The following test illustrates the problem on 7.x:
// put this in DateFieldMapperTests
public void testFormat() throws Exception {
String mapping = Strings.toString(XContentFactory.jsonBuilder().startObject().startObject("type")
.startObject("properties").startObject("@timestamp").field("type", "date")
.field("format", "yyyy-MM-dd'T'HH:mm:ss.SSSZ").endObject().endObject()
.endObject().endObject());
DocumentMapper mapper = parser.parse("type", new CompressedXContent(mapping));
ParsedDocument doc = mapper.parse(new SourceToParse("test", "type", "1", BytesReference
.bytes(XContentFactory.jsonBuilder()
.startObject()
.field("@timestamp", "2013-01-25T20:16:55.000Z")
.endObject()),
XContentType.JSON));
assertNotNull(doc);
}
and the corresponding test on 6.x passes:
public void testFormat() throws Exception {
String mapping = Strings.toString(XContentFactory.jsonBuilder().startObject().startObject("type")
.startObject("properties").startObject("@timestamp").field("type", "date")
.field("format", "yyyy-MM-dd'T'HH:mm:ss.SSSZ").endObject().endObject()
.endObject().endObject());
DocumentMapper mapper = parser.parse("type", new CompressedXContent(mapping));
ParsedDocument doc = mapper.parse(SourceToParse.source("test", "type", "1", BytesReference
.bytes(XContentFactory.jsonBuilder()
.startObject()
.field("@timestamp", "2013-01-25T20:16:55.000Z")
.endObject()),
XContentType.JSON));
assertNotNull(doc);
}
Involved in this investigation: @pheyos and @dolaru (ML QA), @joegallo (Cloud, where rolling upgrades got stuck), @dnhatn (ES debugging + tests)
QA testing of ML functionality in an upgrade from 6.7.1 to 7.0.0-rc2 revealed an issue where moving shards from 6.7.1 to 7.0.0-rc2 nodes were getting stuck in the translog phase. The reason for this was that the 7.0.0-rc2 would not be able to parse the documents from the 6.7.1 node, internally throwing a
MapperParsingExceptionof the formfailed to parse field [@timestamp] of type [date] in document with id 'AVr3KgpQqy1yzebk-FVk'. This would in turn make the peer recovery hang, as the node with theMapperParsingExceptionwould indefinitely wait for a cluster state update with an updated mapping. This issue does not only present a problem for rolling upgrades, but also does not allow any peer recovery of that index in a pure 7.x cluster unless the translog with the corresponding documents is completely purged.The problematic dateformat field in the mappings of the index looked as follows:
The documents were of the form
The problem seems to be an incompatibility between 6.x and 7.x when it comes to date format parsing, possibly related to #27330
The following test illustrates the problem on 7.x:
and the corresponding test on 6.x passes:
Involved in this investigation: @pheyos and @dolaru (ML QA), @joegallo (Cloud, where rolling upgrades got stuck), @dnhatn (ES debugging + tests)