{"id":5708,"date":"2017-11-15T11:25:20","date_gmt":"2017-11-15T19:25:20","guid":{"rendered":"https:\/\/learnopencv.com\/?p=5708"},"modified":"2023-10-03T01:27:58","modified_gmt":"2023-10-03T08:27:58","slug":"understanding-autoencoders-using-tensorflow-python","status":"publish","type":"post","link":"https:\/\/learnopencv.com\/understanding-autoencoders-using-tensorflow-python\/","title":{"rendered":"Autoencoders Explored: Understanding and Implementing Denoising Autoencoders with Tensorflow (Python)"},"content":{"rendered":"\n

In this article, we will learn about autoencoders in deep learning. We will show a practical implementation of using a Denoising Autoencoder on the MNIST<\/a> handwritten digits dataset as an example. In addition, we are sharing an implementation of the idea in Tensorflow.<\/p>\n\n\n\n

1. What is An Autoencoder?<\/h2>\n\n\n\n

An autoencoder is an unsupervised machine-learning algorithm that takes an image as input and reconstructs it using fewer bits. That may sound like image compression, but the biggest difference between an autoencoder and a general purpose image compression algorithm is that in the case of autoencoders, the compression is achieved by learning on a training set of data. While reasonable compression is achieved when an image is similar to the training set used, autoencoders are poor general-purpose image compressors; JPEG compression will do vastly better.<\/p>\n\n\n\n

Autoencoders are similar in spirit to dimensionality reduction techniques like principal component analysis<\/a>. They create a space where the essential parts of the data are preserved while non-essential ( or noisy ) parts are removed.<\/p>\n\n\n\n

There are two parts to an autoencoder<\/p>\n\n\n\n

    \n
  1. Encoder<\/strong>: This is the part of the network that compresses the input into a fewer number of bits. The space represented by these fewer number of bits is called the “latent-space” and the point of maximum compression is called the bottleneck. These compressed bits that represent the original input are together called an “encoding” of the input.<\/li>\n\n\n\n
  2. Decoder<\/strong>: This is the part of the network that reconstructs the input image using the encoding of the image.<\/li>\n<\/ol>\n\n\n\n

    Let’s look at an example to understand the concept better.<\/p>\n\n\n\n

    \"Autoencoder<\/a>
    Figure 1: 2-layer autoencoder<\/figcaption><\/figure>\n\n\n\n\t\t
    \n\t\t\t\t\t
    \n\t\t\t\t\t\t
    \n\t\t\t\t\t
    \n\t\t\t
    \n\t\t\t\t\t\t
    \n\t\t\t\t
    \n\t\t\t\t\t
    \n\t
    \n\t\t
    \n\t\t\t\t\t\t
    \n\t\t\t\t
    \n\t\t\t\t\t\n\t\t\t\t\t\t
    \n\t\t\t\t\t\t\t
    \n\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t
    \n\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\"\"\t\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t

    <\/i> 100K+ Learners
    \n<\/i> 3 Hours of Learning<\/p>\n\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\n\t\t\t\t\t\t\t\t\tJoin Free OpenCV Bootcamp<\/strong>\n\t\t\t\t\t\t\t\t\t<\/span>\n\t\t\t\t\t\t\t\t<\/span>\n\t\t\t\t\t\t\t<\/div>\n\t\t\t\t\t\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t\t<\/a>\n\t\t\t\t<\/div>\n\t\t\t<\/div>\n\t\t\t\t\t\t

    \n\t\t\t\t
    \n\t\t\t\t\t\n\t\t\t\t\t\t
    \n\t\t\t\t\t\t\t
    \n\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t
    \n\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\"\"\t\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t

    <\/i> 15K+ Learners
    \n<\/i> 3 Hours of Learning<\/p>\n\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\n\t\t\t\t\t\t\t\t\tJoin Free TensorFlow Bootcamp<\/strong>\n\t\t\t\t\t\t\t\t\t<\/span>\n\t\t\t\t\t\t\t\t<\/span>\n\t\t\t\t\t\t\t<\/div>\n\t\t\t\t\t\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t\t<\/a>\n\t\t\t\t<\/div>\n\t\t\t<\/div>\n\t\t\t\t\t\t

    \n\t\t\t\t
    \n\t\t\t\t\t\n\t\t\t\t\t\t
    \n\t\t\t\t\t\t\t
    \n\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t
    \n\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\"\"\t\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t

    <\/i> 10K+ Learners
    \n<\/i> 8 Hours of Learning<\/p>\n\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\n\t\t\t\t\t\t\t\t\tJoin Free PyTorch Bootcamp<\/strong>\n\t\t\t\t\t\t\t\t\t<\/span>\n\t\t\t\t\t\t\t\t<\/span>\n\t\t\t\t\t\t\t<\/div>\n\t\t\t\t\t\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t\t<\/a>\n\t\t\t\t<\/div>\n\t\t\t<\/div>\n\t\t\t\t\t<\/div>\n\n\t\t\t\t\t

    <\/div>\n\t\t\n\t\t\t<\/div>\n<\/div>\n<\/div>\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t
    \n\t\t\t\t
    \n\t\t\t\t\t\t\t\t\t
    \n\t\t\t\t\t\n\t\t\t\t\t\t\n\t\t\t\t\t\t\n\t\t\t\t<\/i>\t\t\t<\/span>\n\t\t\t\t\t\t\t\t\tView all AI Free Courses<\/span>\n\t\t\t\t\t<\/span>\n\t\t\t\t\t<\/a>\n\t\t\t\t<\/div>\n\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t\t<\/div>\n\t\t<\/div>\n\t\t\t\t\t<\/div>\n\t\t<\/section>\n\t\t\t\t<\/div>\n\t\t \n\n\n\n

    In the above picture, we show a vanilla autoencoder — a 2-layer autoencoder with one hidden layer. The input and output layers have the same number of neurons. We feed five real values into the autoencoder compressed by the encoder into three real values at the bottleneck (middle layer). Using these three real values, the decoder tries reconstructing the five real values we had fed as input to the network.<\/p>\n\n\n\n

    In practice, there are a far larger number of hidden layers in between the input and the output.<\/p>\n\n\n\n

    There are various kinds of autoencoders like a sparse autoencoder, variational autoencoder<\/a>, and denoising autoencoder. In this post, we will learn about a denoising autoencoder.<\/p>\n\n\n\n

    2. Denoising Autoencoders<\/h2>\n\n\n\n
    \"Denoising
    Figure 2: Denoising autoencoder<\/figcaption><\/figure>\n\n\n\n

    The idea behind a denoising autoencoder is to learn a representation (latent space) that is robust to noise. We add noise to an image and then feed this noisy image as an input to our network. The encoder part of the autoencoder transforms the image into a different space that preserves the handwritten digits but removes the noise. As we will see later, the original image is 28 x 28 x 1 image, and the transformed image is 7 x 7 x 32. You can think of the 7 x 7 x 32 image as a 7 x 7 image with 32 color channels.<\/p>\n\n\n\n

    The decoder part of the network then reconstructs the original image from this 7 x 7 x 32 image, and voila the noise is gone!<\/p>\n\n\n\n

    How does this magic happen?<\/p>\n\n\n\n

    During training, we define a loss (cost function) to minimize the difference between the reconstructed image and the original noise-free image. In other words, we learn a 7 x 7 x 32 space that is noise-free.<\/p>\n\n\n\n

    3. Implementation of Denoising Autoencoder<\/h2>\n\n\n\n

    This implementation is inspired by this excellent post Building Autoencoders in Keras<\/a>.<\/p>\n\n\n\n

    3.1 The Network<\/h3>\n\n\n\n

    The images are matrices of size 28 x 28. We reshape the image to be of size 28 x 28 x 1, convert the resized image matrix to an array, rescale it between 0 and 1, and feed this as an input to the network. The encoder transforms the 28 x 28 x 1 image to a 7 x 7 x 32 image. You can think of this 7 x 7 x 32 image as a point in a 1568 ( because 7 x 7 x 32 = 1568 ) dimensional space. This 1568 dimensional space is called the bottleneck or the latent space. The architecture is graphically shown below.<\/p>\n\n\n\n

    \"Encoder
    Figure 3: Architecture of encoder model<\/figcaption><\/figure><\/div>\n\n\n\n

    The decoder does the exact opposite of an encoder; it transforms this 1568 dimensional vector back to a 28 x 28 x 1 image. We call this output image a “reconstruction” of the original image. The structure of the decoder is shown below.<\/p>\n\n\n\n

    \"Decoder
    Figure 4: Architecture of decoder model<\/figcaption><\/figure><\/div>\n\n\n\n

    Let’s dive into the implementation of an autoencoder using TensorFlow.<\/p>\n\n\n\n

    3.2 Encoder<\/h3>\n\n\n\n

    The encoder has two convolutional layers and two max pooling layers. Both Convolution layer-1 and Convolution layer-2 have 32-3 x 3 filters. There are two max-pooling layers, each of size 2 x 2.<\/p>\n\n\n\n

    \r\n Download Code<\/strong>\r\nTo easily follow along this tutorial, please download code by clicking on the button below. It's FREE!\r\n\r\n
    Download Code<\/a><\/div>\r\n\r\n<\/div>\r\n\t\t
    \n\t\t\t\t\t
    \n\t\t\t\t\t\t
    \n\t\t\t\t\t
    \n\t\t\t
    \n\t\t\t\t\t\t
    \n\t\t\t\t
    \n\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\"\"\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t
    \n\t\t\t\t
    \n\t\t\t\t\t\t\t\t\t Click here to download the source code to this post <\/a>\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t\t<\/div>\n\t\t<\/div>\n\t\t\t\t\t<\/div>\n\t\t<\/section>\n\t\t\t\t<\/div>\n\t\t\n\n\n
    \nencoder = Sequential([\n    # convolution\n    Conv2D(\n        filters=32,\n        kernel_size=(3,3),\n        strides=(1,1),\n        padding='SAME',\n        use_bias=True,\n        activation=lrelu,\n        name='conv1'\n    ),\n    # the input size is 28x28x32\n    MaxPooling2D(\n        pool_size=(2,2),\n        strides=(2,2),\n        name='pool1'\n    ),\n    # the input size is 14x14x32\n    Conv2D(\n        filters=32,\n        kernel_size=(3,3),\n        strides=(1,1),\n        padding='SAME',\n        use_bias=True,\n        activation=lrelu,\n        name='conv2'\n    ),\n    # the input size is 14x14x32\n    MaxPooling2D(\n        pool_size=(2,2),\n        strides=(2,2),\n        name='encoding'\n    )\n    # the output size is 7x7x32\n])\n<\/pre><\/div>\n\n\n
    \"Encoder
    Figure 5: Encoder block diagram<\/figcaption><\/figure><\/div>\n\n\n\n
    3.3 Decoder<\/h3>\n\n\n\n
    The decoder has two Conv2d_transpose layers, two Convolution layers, and one Sigmoid activation function. Conv2d_transpose is for upsampling, which is opposite to the role of a convolution layer. The Conv2d_transpose layer upsamples the compressed image twice each time we use it.<\/p>\n\n\n
    \ndecoder = Sequential([\n    Conv2D(\n        filters=32,\n        kernel_size=(3,3),\n        strides=(1,1),\n        name='conv3',\n        padding='SAME',\n        use_bias=True,\n        activation=lrelu\n    ),\n    # updampling, the input size is 7x7x32\n    Conv2DTranspose(\n        filters=32,\n        kernel_size=3,\n        padding='same',\n        strides=2,\n        name='upsample1'\n    ),\n    # upsampling, the input size is 14x14x32\n    Conv2DTranspose(\n        filters=32,\n        kernel_size=3,\n        padding='same',\n        strides=2,\n        name='upsample2'\n    ),\n    # the input size is 28x28x32\n    Conv2D(\n        filters=1,\n        kernel_size=(3,3),\n        strides=(1,1),\n        name='logits',\n        padding='SAME',\n        use_bias=True\n    )    \n])\n<\/pre><\/div>\n\n\n
    \"Decoder
    Figure  6: Decoder Block Diagram<\/figcaption><\/figure>\n\n\n\n
    The resultant encoder-decoder model class is represented as:<\/p>\n\n\n
    \n# model class definition\nclass EncoderDecoderModel(Model):\n    def __init__(self, is_sigmoid=False):\n        super(EncoderDecoderModel, self).__init__()\n        # assign encoder sequence\n        self._encoder = encoder\n        # assign decoder sequence \n        self._decoder = decoder\n        self._is_sigmoid = is_sigmoid\n        \n    # forward pass\n    def call(self, x):\n        x = self._encoder(x)\n        decoded = self._decoder(x)\n        if self._is_sigmoid:\n            decoded = tf.keras.activations.sigmoid(decoded)\n        return decoded\n<\/pre><\/div>\n\n\n
    Finally, we calculate the loss of the output using cross-entropy<\/a> loss function and use Adam optimizer<\/a> to optimize our loss function.<\/p>\n\n\n\n
    3.4 Why do we use a leaky ReLU and not a ReLU as an activation function?<\/h3>\n\n\n\n
    We want gradients to flow while we backpropagate through the network. We stack many layers in a system in which there are some neurons whose value drop to zero or become negative. Using a ReLU as an activation function clips the negative values to zero and in the backward pass, the gradients do not flow through those neurons where the values become zero. Because of this the weights do not get updated, and the network stops learning for those values. So using ReLU is not always a good idea. However, we encourage you to change the activation function to ReLU and see the difference.<\/p>\n\n\n
    \n# define leaky ReLU function\ndef lrelu(x, alpha=0.1):\n    return tf.math.maximum(alpha*x, x)\n<\/pre><\/div>\n\n\n
    Therefore, we use a leaky ReLU which instead of clipping the negative values to zero, cuts them to a specific amount based on a hyperparameter alpha. This ensures that the network learns something even when the pixel value is below zero.<\/p>\n\n\n\n
    3.5 Load the data<\/h3>\n\n\n\n
    Once the architecture has been defined, we load the training and validation data.<\/p>\n\n\n\n
    As shown below, Tensorflow allows us to easily load the MNIST data. The training and testing data loaded is stored in variables train_imgs<\/code> and test_imgs<\/code> respectively. Since it’s an unsupervised task we do not care about the labels.<\/p>\n\n\n
    \n# load mnist dataset\n(train_imgs, train_labels), (test_imgs, test_labels) = tf.keras.datasets.mnist.load_data()\n\n# fit image pixel values from 0 to 1\ntrain_imgs, test_imgs = train_imgs \/ 255.0, test_imgs \/ 255.0 \n<\/pre><\/div>\n\n\n
    3.6 Data Analysis<\/h3>\n\n\n\n
    Before training a neural network<\/a>, it is always a good idea to do a sanity check on the data.<\/p>\n\n\n\n
    Let\u2019s see how the data looks like. The data consists of handwritten numbers ranging from 0 to 9, along with their ground truth labels. It has 55,000 train samples and 10,000 test samples. Each sample is a 28\u00d728 grayscale image. Let’s view the data details:<\/p>\n\n\n
    \n# check data array shapes:\nprint(\"Size of train images: {}, Number of train images: {}\".format(train_imgs.shape[-2:], train_imgs.shape[0]))\nprint(\"Size of test images: {}, Number of test images: {}\".format(test_imgs.shape[-2:], test_imgs.shape[0]))\n<\/pre><\/div>\n\n\n
    The output<\/strong> is:<\/p>\n\n\n
    \nSize of train images: (28, 28), Number of train images: 60000\nSize of test images: (28, 28), Number of test images: 10000\n<\/pre><\/div>\n\n\n
    The visualization of train and test image examples:<\/p>\n\n\n
    \n# plot image example from training images\nplt.imshow(train_imgs[1], cmap='Greys')\nplt.show()\n\n# plot image example from test images\nplt.imshow(test_imgs[0], cmap='Greys')\nplt.show()\nplt.close()\n<\/pre><\/div>\n\n\n
    Output:<\/strong><\/p>\n\n\n\n
    \"Train
    Figure 7: Train and test MNIST images<\/figcaption><\/figure><\/div>\n\n\n\n
    3.7 Preprocessing the data<\/h3>\n\n\n\n
    The images are grayscale and the pixel values range from 0 to 255. We apply the following preprocessing to the data before feeding it to the network.<\/p>\n\n\n\n
      \n
    1. Add a new dimension to the train and test images, which will be fed into the network.<\/li>\n<\/ol>\n\n\n
      \n# prepare training reference images: add new dimension\ntrain_imgs_data = train_imgs[..., tf.newaxis]\n\n# prepare test reference images: add new dimension\ntest_imgs_data = test_imgs[..., tf.newaxis]\n<\/pre><\/div>\n\n\n
        \n
      1. Add noise to both train and test images which we then feed into the network. The noise factor is a hyperparamter and can be tuned accordingly.<\/li>\n<\/ol>\n\n\n
        \n# add noise to the images for train and test cases\ndef distort_image(input_imgs, noise_factor=0.5):\n    noisy_imgs = input_imgs + noise_factor * np.random.normal(loc=0.0, scale=1.0, size=input_imgs.shape) \n    noisy_imgs = np.clip(noisy_imgs, 0., 1.)\n    return noisy_imgs\n\n# prepare distorted input data for training\ntrain_noisy_imgs = distort_image(train_imgs_data)\n\n# prepare distorted input data for evaluation\ntest_noisy_imgs = distort_image(test_imgs_data)\n<\/pre><\/div>\n\n\n
        Let’s illustrate the noisy images :<\/p>\n\n\n
        \n# plot distorted image example from training images\nimage_id_to_plot = 0\nplt.imshow(tf.squeeze(train_noisy_imgs[image_id_to_plot]), cmap='Greys')\nplt.title(\"The number is: {}\".format(train_labels[image_id_to_plot]))\nplt.show()\n\n# plot distorted image example from test images\nplt.imshow(tf.squeeze(test_noisy_imgs[image_id_to_plot]), cmap='Greys')\nplt.title(\"The number is: {}\".format(test_labels[image_id_to_plot]))\nplt.show()\nplt.close()\n<\/pre><\/div>\n\n\n
        Output:<\/strong><\/p>\n\n\n\n
        \"Noisy
        Figure 8: Noisy train and test MNIST images<\/figcaption><\/figure><\/div>\n\n\n\n
        3.8 Train and evaluate the model<\/h3>\n\n\n\n
        The network is ready to get trained. We specify the number of epochs as 25 with batch size of 64. This means that the whole dataset will be fed to the network 25 times. We will be using the test data for validation. <\/p>\n\n\n
        \n# define custom target function for further minimization\ndef cost_function(labels=None, logits=None, name=None):\n    loss = tf.nn.sigmoid_cross_entropy_with_logits(labels=labels, logits=logits, name=name)\n    return tf.reduce_mean(loss)\n\n# init the model\nencoder_decoder_model = EncoderDecoderModel()\n\n# training loop params\nnum_epochs = 25\nbatch_size_to_set = 64\n\n# training process params\nlearning_rate = 1e-5\n# default number of workers for training process\nnum_workers = 2\n\n# initialize the training configurations such as optimizer, loss function and accuracy metrics\nencoder_decoder_model.compile(optimizer=tf.compat.v1.train.AdamOptimizer(learning_rate=learning_rate),loss=cost_function,metrics=None)\n\nresults = encoder_decoder_model.fit(\n    train_noisy_imgs,\n    train_imgs_data,\n    epochs=num_epochs,\n    batch_size=batch_size_to_set,\n    validation_data=(test_noisy_imgs, test_imgs_data),\n    workers=num_workers,\n    shuffle=True\n)\n<\/pre><\/div>\n\n\n
        After 25 epochs, we can see our training loss and validation loss is quite low, which means our network did a pretty good job. Let’s now see the loss plot between training and validation data using the introduced utility function plot_losses(results)<\/code>.<\/p>\n\n\n\n
        3.10 Training Vs. Validation Loss Plot<\/h3>\n\n\n\n
        We’ve defined the utility function for plotting the losses:<\/p>\n\n\n
        \n# funstion for train and val losses visualizations\ndef plot_losses(results):\n    plt.plot(results.history['loss'], 'bo', label='Training loss')\n    plt.plot(results.history['val_loss'], 'r', label='Validation loss')\n    plt.title('Training and validation loss',fontsize=14)\n    plt.xlabel('Epochs ',fontsize=14)\n    plt.ylabel('Loss',fontsize=14)\n    plt.legend()\n    plt.show()\n    plt.close()\n\n# visualize train and val losses\nplot_losses(results)\n<\/pre><\/div>\n\n\n
        The result is:<\/p>\n\n\n\n
        \"Training
        Figure 9: Training and validation losses<\/figcaption><\/figure><\/div>\n\n\n\n
        From the above loss plot, we can observe that the validation loss and training loss steadily decrease in the first ten epochs. This training loss and the validation loss are also very close to each other. This means that our model has generalized well to unseen test data.<\/p>\n\n\n\n
        We can further validate our results by observing the original, noisy and reconstruction of test images.<\/p>\n\n\n\n
        3.11 Results<\/h3>\n\n\n\n
        \"Original,
        Figure 10: Representation of MNIST images on different stages<\/figcaption><\/figure><\/div>\n\n\n\n
        From the above figures, we can observe that our model did a good job in denoising the noisy images that we had fed into our model.<\/p>\n\n\n","protected":false},"excerpt":{"rendered":"
        In this article, we will learn about autoencoders in deep learning. We will show a practical implementation of using a Denoising Autoencoder on the MNIST handwritten digits dataset as an example. In addition, we are sharing an implementation of the idea in Tensorflow. 1. What is An Autoencoder? An autoencoder is an unsupervised machine-learning algorithm […]<\/p>\n","protected":false},"author":13,"featured_media":6108,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"_acf_changed":false,"footnotes":""},"categories":[56,454,455,104],"tags":[245,2341,1941,2340,101,246,1942,220,211,2339,866],"coauthors":[445,528],"class_list":["post-5708","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-deep-learning","category-keras","category-tensorflow","category-tutorial","tag-autoencoder","tag-autoencoder-neural-network","tag-autoencoders-in-deep-learning","tag-autoencoders-python","tag-convolutional-neural-network","tag-denoising","tag-denoising-autoencoders-and-where-to-find-them","tag-mnist","tag-tensorflow","tag-unsupervised-learning","tag-variational-autoencoder"],"acf":[],"aioseo_notices":[],"featured_image_src":"https:\/\/learnopencv.com\/wp-content\/uploads\/2017\/11\/denoising-autoencoder.jpg","featured_image_src_square":"https:\/\/learnopencv.com\/wp-content\/uploads\/2017\/11\/denoising-autoencoder.jpg","author_info":{"display_name":"Aditya Sharma","author_link":"https:\/\/learnopencv.com\/author\/adityasharma\/"},"aioseo_head":"\n\t\t\n\t\n\t\n\t\n\t\n\t\n\t\n\t\n\t\t\n\t\t\n\t\t\n\t\t\n\t\t\n\t\t\n\t\t\n\t\t\n\t\t\n\t\t\n\t\t\n\t\t\n\t\t\n\t\t\n\t\t\n\t\t\n\t\t\n\t\t\n\t\t\n\t\t