A few interesting articles
A detailed tutorial on how to build a generic TCP proxy and a well written article explaining how Redis works and how it is commonly used .
Hello,
I know it has been a while since the last issue, and in case you are clueless, this is a newsletter where I share a few interesting reads that I find online. Most of them are usually technical but there can be some variations from time to time. Hope that is something that you are still interested in.
Here are two articles that I really enjoyed reading through last week.
How to build a TCP proxy (4-part series)
I work at Requestly which is an easy-to-use proxy for frontend developers. Think of it like burp but for QA engineers rather than bug hunters. So I am always eager to learn more about proxies.
Tools like Burp, Requestly, Charles Proxy, etc. all act as HTTP proxies (layer 7 proxies). This is usually enough for most use cases because most of the web that we use day to day is built on top of HTTP.
To intercept your HTTP traffic with any of these proxies, you need to add the proxy’s IP and port to your system/app’s proxy settings. Even if you are integrating it inside your source code, most HTTP clients like Axios (for NodeJS) or RetroFit (for android) accept a proxy attribute to alter this setting across your app with just a few lines of code (Basically, you are changing the destination for all your requests)
In the case of a TCP (layer 4) proxy, you need to find a different way to redirect your requests to the proxy.
This article goes through all the details of building a generic TCP proxy and provides an interesting solution for forwarding all your TCP traffic to a proxy.
The author does so by creating a simple fake DNS server, running it along with the proxy and changing the default Domain server that his mobile uses to resolve domain names (which seems surprisingly easy to do).
We know that every request on the web first goes to a DNS server so that we can resolve easily recognisable URLs like “nsrcodes.com” to their IP address (which is what your browser needs to make the request).
Now, if you can control which IPs the domain names map to, you have essentially got control of redirecting any request to wherever you wish to send it, which in our case is our PROXY server.
So, all the requests first go to this fake DNS server and this server decides whether it wants to send these requests to the proxy or hit an actual DNS server and return a response. Genius!
Along with the implementation for this, the article also contains simple-to-follow explanations on how to build a TCP proxy from scratch.
Redis Explained
For those unfamiliar with Redis, it is a general-purpose database that stores data directly inside the main memory
I recently had the chance of using Redis inside our infrastructure. We used it as a simple time-based cache for frequently accessed data. Since then I have been eager to know what else Redis is capable of and more importantly, how it actually works “in-memory”
The problem with an in-memory database is how you handle the persistence of data. Since the data is all in memory, shutting the system down (either because of faults; which are bound to happen; or just regular redeployment) will lead to loss of data. Redis handles this quite well with an async process that runs periodically to save snapshots or append-only files (file with a log of all write operations on the database for easy recreation of the current state).
My favourite part is at the end where it describes how Redis (which is inherently single-threaded) spawns forks of its main process to run this async persistence workflow (I had no clue how these two processes would share the addresses of the complete memory, and not cause conflicts)
Another problem is the hardware limitation. Since this is an in-memory database, you can only have so much ram. The obvious answer is to shard the data across several nodes, but since I had never actually worked with databases that much, I never understood how sharding works. This article explains how Redis handles sharding when deployed as a cluster of Redis nodes
Hope this was useful, let me know your thoughts on anything that you find interesting from the above two articles. Until then…
