You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Copy file name to clipboardExpand all lines: docs/src/_getting-started/part4.md
+6-6Lines changed: 6 additions & 6 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -1,5 +1,5 @@
1
1
---
2
-
title: Part 4 - Inline Data Mapping, E(t)LT
2
+
title: Part 4 - Inline Data Mapping
3
3
description: Part 4 - If you're ready to get started with Meltano and run an EL[T] pipeline with a data source and destination of your choosing, you've come to the right place!
4
4
layout: doc
5
5
weight: 4
@@ -8,11 +8,11 @@ weight: 4
8
8
9
9
Let’s learn by example.
10
10
11
-
Throughout this tutorial, we’ll walk you through the creation of a end-to-end modern E(t)LT stack.
11
+
Throughout this tutorial, we’ll walk you through the creation of a end-to-end modern ELT stack.
12
12
13
13
In parts [1](/getting-started/part1), [2](/getting-started/part2), and [3](/getting-started/part3) we built an ELT pipeline. We took all the data from the commits on one repository at GitHub and extracted the authors working on it. However now we realized, we stored a lot of information where we really might want to hide a few of those pieces.
14
14
15
-
We're going to do light-weight transformations ("little t or (t)"), also called **"inline data mappings"** to clean up the data before storing them anywhere. We will use these inline data mappings to hide all emails inside the JSON blob we receive. In the Meltano world, these data mappings are also called [stream maps](https://sdk.meltano.com/en/latest/stream_maps.html).
15
+
We're going to do light-weight transformations, also called **"inline data mappings"** to clean up the data before storing them anywhere. We will use these inline data mappings to hide all emails inside the JSON blob we receive. In the Meltano world, these data mappings are also called [stream maps](https://sdk.meltano.com/en/latest/stream_maps.html).
16
16
17
17
<divclass="notification is-success">
18
18
<p>If you're having trouble throughout this tutorial, you can always head over to the <a href="https://meltano.com/slack">Slack channel</a> to get help.</p>
@@ -81,7 +81,7 @@ These lines define the name "hide-github-mails" as the name of our mapping. We c
81
81
```
82
82
These lines define one transformation. We instruct to target the stream "commits", and therein the field "commit". We then use the field paths to navigate to the two emails we know are contained within this message and set the type to "HASH". Using "HASH" means we will still be able to tell whether two emails are the same, but not be able to read the email. They will be replaced with a SHA-256 hash of the email.
83
83
84
-
## Run the data integration (E(t)LT) pipeline
84
+
## Run the data integration pipeline
85
85
Now we're ready to run the data integration process with these modifications again. To do so, we'll need to clean up first, since we already ran the EL process in part 1. The primary key is still the same and as such the ingestion would fail.
86
86
87
87
Drop the table inside your local postgres by running a docker exec:
@@ -90,7 +90,7 @@ Drop the table inside your local postgres by running a docker exec:
If everything was configured correctly, you should now see your data flow from your source into your destination! Take your favourite SQL tool, connect to the database using the connection details set and check the table `commits` inside the schema `tap_github`. The JSON blob inside the column `commit` should now contain no e-mail adresses but rather the hashed values for both fields.
123
123
124
124
## Next Steps
125
-
There we have it, a complete E(t)LT pipeline, congratulations!
125
+
There we have it, a complete ELT pipeline with inline data mappings, congratulations!
126
126
127
127
One last thing for you to do: try to run the following command to celebrate:
0 commit comments