Having introduced mapping, we will now go into a bit of more detail about data types.\u00a0There are four categories of data types in Elasticsearch, namely core data types, complex data types, geo data types and specialized data types, which we
\nwill all take a look at now. Feel free to skip this article and move onto the next one if you are already familiar with the various data types.<\/p>\n
The first category that I will talk about is core data types, beginning with strings.<\/p>\n
Not surprisingly string fields accept string values. This data type can be sub-divided into full text and keywords, which I will discuss now.<\/p>\n
Full text fields are typically used for text based relevance searches such as when searching for products by name. These fields are analyzed, which means that data is passed through an analyzer to convert strings into a list of individual terms. This happens before the data gets indexed and is what enables Elasticsearch to search for individual words within a full text field. Full text fields are not used for sorting and are rarely used for aggregations.<\/p>\n
The keywords type is used for storing exact values such as tags, statuses and such. The exact string value is added to the index as a single term, because keyword fields are not analyzed. Fields of this type are typically used for filtering, such as finding all products that are on sale. They are also used for sorting and aggregations quite often.<\/p>\n
The next core data type is the numeric data type. As you can probably tell, this data type is used for storing numeric values such as integers, floats and doubles, e.g. 1<\/span> and 1.5<\/span>. The numeric data type supports the following numeric types.<\/p>\n There is also a “date” data type in Elasticsearch. Dates can be either a string containing formatted dates (e.g. 2016-01-01<\/span> or 2016\/01\/01 12:00:00<\/span>), an integer representing the seconds since the epoch or a long number representing the milliseconds since the epoch. The latter is how Elasticsearch stores dates internally.<\/p>\n When dealing with dates, it\u00a0is\u00a0possible to choose their formats. The default format is defined as a date with an optional time or the number of milliseconds since the epoch. This is defined as follows:\u00a0strict_date_optional_time||epoch_millis<\/span>. Below are a few examples of some of the default formats that you can use when adding dates.<\/p>\n Note that the default date parser supports formats defined by the ISO standard<\/a>, so these examples are not the only formats that you can use. However, I would recommend using these as a convention within your project. If you define your own date format, then you can use various constant values that Elasticsearch provides, which act as placeholders when defining the format. You can find a full list of these in the documentation<\/a>. Note that you can use the boolean OR<\/span> as in a programming language, by separating two placeholders with two pipes (||<\/span>).<\/p>\n If a field can only contain on<\/span> or off<\/span> values, then the boolean data type is appropriate. Just like in JSON, boolean fields can accept true<\/span> and\u00a0false<\/span> values. They also accept strings and numbers, which will be interpreted as a boolean. I would personally not recommend this, unless you need to\u00a0do this for some reason. False values include the boolean false<\/span>, a string with a value of “false”<\/span>, “off”<\/span>, “no”<\/span>, “0”<\/span> or an empty string (“”<\/span>), or the number 0<\/span>.\u00a0Everything that is not interpreted as false<\/span>, will be interpreted as true<\/span>.<\/p>\n It’s also possible to store binary data within a field, such as a file. This data will be stored as a Base64 encoded string and will not be searchable. An example is the string\u00a0aHR0cDovL2NvZGluZ2V4cGxhaW5lZC5jb20=<\/span>.<\/p>\n Next, we will discuss complex data types.<\/p>\n JSON documents are hierarchical, meaning that an object may container inner objects. In Elasticsearch, these are flattened before being indexed and are\u00a0stored as simple key\/value pairs. Consider the following\u00a0example.<\/p>\n If customer<\/span> is an object that has an age<\/span> property, then this property will be\u00a0stored within a field named customer.age<\/span>. The same applies for nested objects as you can see with the address<\/span> object.<\/p>\n Often times, you might want to store an array of values. You might be surprised to hear that Elasticsearch does not have an array type. This is because\u00a0any field can contain zero or more values by default, without having to explicitly define this. This means that by simply passing an array of values to\u00a0a field, Elasticsearch will automatically store these values, even for a field that has been mapped as a string, for example. Do, however, note that all\u00a0of the values within an array must be of the same data type. If no explicit mapping defines the data type, then Elasticsearch will infer it based on the\u00a0first value in the array.<\/p>\n Below, you can see a few examples of adding arrays to an Elasticsearch field.<\/p>\n Most noticeable is the\u00a0array of arrays that is flattened into a single array of values. Besides the simple data types, it is also possible to add complex data types to an array,\u00a0such as objects.<\/p>\n Above,\u00a0I showed you an example of an array of objects. However, this does not work as you might expect. As I told you a minute ago,\u00a0Elasticsearch flattens object hierarchies into key\/value pairs because Lucene has no concept of inner objects like JSON does. This means that you cannot\u00a0query each object independently of the other objects in the array.\u00a0Please consider the below example.<\/p>\n This array of objects is stored similar to this:<\/p>\n An array contains two user\u00a0objects with the name<\/span> and age<\/span> properties. When this array is stored in Lucene, all of the values for a given property in the objects are added as an\u00a0array into a single field – in this case fields named users.name<\/span> and users.age<\/span>. In doing so, the associations between the properties are lost. In\u00a0this particular example, the association between the name Andy<\/span> and the age 26<\/span> is lost, because the values are not sorted. The consequence is that a\u00a0search for a user named Andy<\/span> who is 26<\/span> years old, would return incorrect results. The good news is that there is a solution to this problem, in case\u00a0you need to be able to perform such a search.<\/p>\n The solution to the problem mentioned above is the nested<\/span> data type. This data type should be used when you need to index arrays of objects and\u00a0maintain the independence of each object. This means that the values for all of the objects will not be mixed together as you saw above.\u00a0Internally, each object in the array is indexed as a separate hidden document. The result of using this data type is that you can query for specific\u00a0objects based on their field values by using a special query named nested<\/span>. What happens internally in Elasticsearch, is that the nested objects are\u00a0queried as if they were indexed as separate documents. This\u00a0is in fact the case technically, but you do not actually need to be aware of this.<\/p>\n Next, we will briefly walk through some of the\u00a0geo data types, which are unsurprisingly used for storing geographical data.<\/p>\n The geo-point<\/span> data type is used for storing latitude-longitude pairs and can be used for searching for documents within a radius of some coordinates,\u00a0sorting by distance from some coordinates, etc. Values for a geo-point field can be added in four different formats.\u00a0The first one is as an object with\u00a0the lat<\/span> and lon<\/span> properties.<\/p>\n The second is as a string with the latitude and longitude separated by a comma, and in that order.<\/p>\n The third option is a\u00a0geohash, which I am not going to explain further.<\/p>\n And the fourth way is as an array.<\/p>\n Notice that compared to the other formats, the latitude and longitude\u00a0have been switched around. The reason for this is to conform with GeoJSON<\/a>.<\/p>\n A geo shape field can contain shapes such as rectangles and polygons, and can therefore be used to construct complex shapes. Therefore, this data type\u00a0should be used when a geo point is not sufficient. Next, we will briefly discuss\u00a0some specialized data types whose use cases are quite narrow.<\/p>\n The first one is the IPv4 data type. Unsurprisingly, it is used to map IPv4 addresses. Elasticsearch stores the values for an IPv4 field as long<\/span> values\u00a0internally.<\/p>\n There is also a data type that is used for auto-complete functionality, named completion<\/span>. It is a so-called prefix suggester and although it does not do\u00a0spell correction, it is useful for providing the user with suggestions while searching, such as on Google. The way is does this, is by storing a finite\u00a0state transducer as part of the index, which allows for very fast loads and executions. However, as long as you know when to use this type, then this is\u00a0not important.<\/p>\n The token_count<\/span> data type is an integer field which accepts string values. This may sound strange, but what happens is that the string values are\u00a0analyzed, and the number of tokens in the strings are indexed. An example use case would be that a name<\/span> property has a length<\/span> field of the\u00a0token_count<\/span> type. A search query could then be executed to find persons whose name contains a given number of tokens, split by space for instance.<\/p>\n There is also an attachment<\/span> data type, which lets Elasticsearch index attachments for many common formats such as PDF files, spreadsheets, PowerPoint\u00a0presentations and more. The attachments are stored as a Base64 encoded string. This functionality is available as a plugin that must be installed on all\u00a0nodes. Doing so is easy and can be done by typing in the below command, assuming that the node is running in a UNIX environment.<\/p>\n If you do\u00a0install this plugin, then note that you must restart each node after the installation has completed.<\/p>\n That’s it! That was a walkthrough of the most important data types in Elasticsearch that you are most likely to need or encounter.<\/p>\n","protected":false},"excerpt":{"rendered":"\n
Dates<\/h3>\n
Dates – Formats<\/h3>\n
\n
Boolean<\/h3>\n
Binary<\/h3>\n
Complex Data Types<\/h2>\n
Object<\/h3>\n
{\r\n\t\"message\": \"Some text...\",\r\n\t\"customer.age\": 26,\r\n\t\"customer.address.city\": \"Copenhagen\",\r\n\t\"customer.address.country\": \"Denmark\"\r\n}<\/code><\/pre>\nArray<\/h3>\n
\n
Arrays – Objects<\/h3>\n
{ \"users : [{ \"name\": \"Andy\", \"age\": 26 }, { \"name\": \"Brenda\", \"age\": 32 }] }<\/code><\/pre>\n{ \"users.name\": [\"Andy\", \"Brenda\"], \"users.age\": [32, 26] }<\/code><\/pre>\nNested<\/h3>\n
Geo Data Types<\/h2>\n
Geo-point<\/h3>\n
{\r\n\t\"location\": { \r\n\t\t\"lat\": 33.5206608,\r\n\t\t\"lon\": -86.8024900\r\n\t}\r\n}<\/code><\/pre>\n{\r\n\t\"location\": \"33.5206608,-86.8024900\" \r\n}<\/code><\/pre>\n{\r\n\t\"location\": \"drm3btev3e86\" \r\n}<\/code><\/pre>\n{\r\n\t\"location\": [-86.8024900,33.5206608] \r\n}<\/code><\/pre>\nGeo-shape<\/h3>\n
\nA geo shape can be constructed in many ways, some of which include adding it as a linestring or as a\u00a0polygon. A linestring is an array of two or more position, which is essentially an array of arrays. If the array only contains two positions, then the\u00a0result is a straight line. A polygon is also added as an array of arrays, where each array contains points. It is important to note that the first and\u00a0last points in the outer array must be the same, which closes the polygon.<\/p>\nSpecialized Data Types<\/h2>\n
IPv4<\/h3>\n
Completion<\/h3>\n
Token\u00a0Count<\/h3>\n
Attachment<\/h3>\n
sudo \/path\/to\/elasticsearchbin\/plugin install mapper-attachments<\/code><\/pre>\n