Have you ever wondered when navigating Google Maps how it comes up with the quickest route to your destination? The answer to your curiosity lies in the magnificent world of weighted graphs and Dijkstra's algorithm.
\nLet's explore this further. Imagine you're traveling on different roads of unequal lengths and different speed limits, thus affecting the time spent on each route. In graph theory, this scenario could symbolize a weighted graph where each edge (or path) carries its weight (the time spent on each road). Therefore, finding the quickest route essentially means figuring out the shortest path in this weighted graph, and here is where Dijkstra's algorithm comes to play. This article will walk you through a deep understanding of weighted graphs and the backbone of quickest route algorithms, Dijkstra's algorithm.
\nLet's explore the following weighted graph.
\n![\"Weighted](\"/_next/static/media/weighted-graph.54f87fa9.svg\"/)
\nThis graph shows two possible routes from the Origin to Destiny. The numbers represent the time it takes to move from one point to another. If you recall from our discussion on breadth-first search, it would suggest the shortest route as Origin -> A -> Destiny. While this path has fewer segments, the quickest option would be through Dijkstra's Algorithm, taking the path Origin -> B -> A -> Destiny, taking a total of 6 minutes.
![\"Weighted](\"/_next/static/media/weighted-graph-solution.3019bf22.svg\"/)
\nNow that we know the result, let's break it down into four simple steps:
\n| Vertex | Time |
|---|---|
| A | 6 |
| B | 2 |
| Destiny | Infinity |
You're at the Origin contemplating your next stop: Node A takes 6 minutes, while Node B takes 2 minutes. The time required to get to the Destiny is still unknown, hence we consider it infinite for now. Therefore, Node B becomes our cheapest node.
\n| Vertex | Time |
|---|---|
| A | |
| B | 2 |
| Destiny | 7 |
From Node B, you realize you could reach Node A in 5 minutes (2 to get to B then 3 to A) which is a minute less than going directly. Subsequently, you update the time for Node A. Similarly, you understand that reaching the Destiny would take 7 minutes (2 to B then 5 to Destiny).
\nThe process is then repeated for Node A and the final path calculated. The quickest route ends up being Origin -> B -> A -> Destiny, taking a total of 6 minutes.
| Vertex | Time |
|---|---|
| A | 5 |
| B | 2 |
| Destiny |
In Dijkstra's Algorithm, each edge possesses an associated number known as the weight and the graph becomes known as a "weighted graph." Whereas previously in our breadth-first search, the "shortest path" referred to the minimum number of segments, in Dijkstra's interpretation, it implies the path with the least cumulative weight.
\nOne caveat in Dijkstra's algorithm is its inefficiency in dealing with cycles. A cycle occurs when you can start at one vertex and traverse through the graph to return to the same vertex. Take a look at the following example:
\n![\"Weighted](\"/_next/static/media/weighted-graph-cycles.f0da8edd.svg\"/)
\nHere, even though the quickest path is Origin -> A -> B -> Destiny taking 3 minutes, Dijkstra's algorithm may keep circling between A and B.
Dijkstra's algorithm fails to handle negative weights. In cases where there are negative weights, like this:
\n![\"Weighted](\"/_next/static/media/weighted-graph-negative.ea08babd.svg\"/)
\nThe quickest path would be Origin -> A -> B -> Destiny, taking 33 minutes. Using Dijkstra's algorithm, we would unusedly arrive at longer routes. Instead, for such cases, we use the Bellman-Ford algorithm, a more suitable algorithm for graphs involving negative weights. While Bellman-Ford isn't in the scope of this article, we'll surely approach it in possible future articles.
Let's dive into creating Dijkstra's algorithm for our initial graph using JavaScript. Before implementing, you'll need three hash tables.
\nThe first table will store the graph, the second table will store the costs, and the third table will store the parents. The cost and parents tables will be updated as the algorithm proceeds. Below you'll find a sample implementation of Dijkstra's algorithm.
\nFirst, let's implement the graph.
\nconst graph = {}\ngraph['origin'] = {}\ngraph['origin']['a'] = 6\ngraph['origin']['b'] = 2\ngraph['a'] = {}\ngraph['a']['destiny'] = 1\ngraph['b'] = {}\ngraph['b']['a'] = 3\ngraph['b']['destiny'] = 5\ngraph['destiny'] = {} // The destination node has no neighbors\nIn order to get all the neighbors of a node, you can use the Object.keys() method.
Now, you need a hash table to store the weights.
\nconst weights = {}\nweights['a'] = 6\nweights['b'] = 2\nweights['destiny'] = Infinity // The time to reach the destination is unknown for now\nAnd the last, but not least, you need a hash table to store the parents.
\nconst parents = {}\nparents['a'] = 'origin'\nparents['b'] = 'origin'\nparents['destiny'] = null // The destination has no parent yet\nFinally, you need a way to keep track of all the processed vertices, as they don't need to be processed more than once.
\nconst processed = new Map()\nNow, with all the necessary data structures in place, you can implement the algorithm.
\nDiagramming the quickest route on Google Maps no longer feels like a giant enigma. In summary, Dijkstra's algorithm aids in navigating a weighted graph efficiently by finding the path with the least total weight. Remember, weighted graphs are your friend when you're trying to find the shortest route in terms of weight or cost, not necessarily the number of segments!
\nHave any more doubts about Dijkstra's Algorithm, weighted graphs, or any other algorithm? Feel free to reach out. You can also subscribe to my newsletter for updates about my latest articles and guides. Happy coding!
", "url": "https://jopcmelo.dev/articles/dijkstras-algorithms", "title": "Understanding Weighted Graphs: Dijkstra's Algorithm Revealed", "summary": "In this guide for developers, we delve into an understanding of weighted graphs and Dijkstra's Algorithm, a popular method for finding the shortest paths in such structures. Master your graph-based problems with this in-depth walkthrough.", "date_modified": "2024-05-05T00:00:00.000Z", "author": { "name": "João Melo" } }, { "id": "https://jopcmelo.dev/articles/introduction-to-graphs", "content_html": "In the world of software engineering, finding the most efficient path between two points is a common challenge, one that can take many forms. Whether it's an AI algorithm strategizing the fewest moves to victory in a game of checkers, a spell-checker computing the minimum number of edits to convert a misspelled word into a correct one, or locating the nearest in-network doctor, the underlying issue is the same. This is where the concept of the shortest path, or the minimum path problem, comes into play.
\nLet's bring this into a real-world example. Imagine you're in San Francisco, looking to travel from Twin Peaks to the Golden Gate Bridge. You plan to take the bus, aiming to minimize the number of transfers. What algorithm could you use to find the route with the fewest steps?
\n![\"Bus](\"/_next/image?url=%2F_next%2Fstatic%2Fmedia%2Fbus-example.2bc42c19.png&w=3840&q=75\"/)
\nYou'd start by considering whether it's possible to reach your destination in one step. Here are all the locations you can get to in a single step:
\n![\"Single](\"/_next/image?url=%2F_next%2Fstatic%2Fmedia%2Fone-step-only.f5ec3eb6.png&w=3840&q=75\"/)
\nThe bridge isn't reachable in one step, so what about two steps?
\n![\"Two](\"/_next/image?url=%2F_next%2Fstatic%2Fmedia%2Ftwo-steps-only.ece9590f.png&w=3840&q=75\"/)
\nOnce again, the bridge eludes us. But what happens at three steps?
\n![\"Three](\"/_next/image?url=%2F_next%2Fstatic%2Fmedia%2Fthree-steps-only.a6ad0b42.png&w=3840&q=75\"/)
\nAha! Now the bridge is highlighted in red, indicating that it's possible to reach in three steps.
\n![\"Right](\"/_next/image?url=%2F_next%2Fstatic%2Fmedia%2Fright-path.9da52882.png&w=3840&q=75\"/)
\nThere may be other routes to the bridge, but they're longer. The algorithm has determined that the shortest path to the bridge requires three steps. This type of problem is solved by an algorithm called Breadth-First Search (BFS), which systematically explores the graph to find the shortest path between two points.
\nTo understand how to go from Twin Peaks to the Golden Gate Bridge using BFS, we need to:
\nLet's delve into what graphs are and how BFS can help us find the shortest path.
\nA graph is a set of connections. It's a powerful model used to represent and solve problems involving networks of points, which in graph terminology are called nodes or vertices, connected by lines known as edges.
\nFor example, consider a scenario where you and your friends are playing poker, and you want to keep track of who owes money to whom. You might say, "João owes money to Juan".
\n![\"A](\"/_next/image?url=%2F_next%2Fstatic%2Fmedia%2Fjoao-juan.7bcf1fd9.png&w=1080&q=75\"/)
\nA graph consists of vertices (like João and Juan) and edges (the relationship that represents who owes money to whom).
\n![\"Vertex](\"/_next/image?url=%2F_next%2Fstatic%2Fmedia%2Fvertex-edges.e9134344.png&w=1080&q=75\"/)
\nThat's essentially it—a graph is simply a collection of vertices connected by edges. A vertex can be directly connected to many other vertices, which are referred to as its neighbors. Graphs provide a way to model how different events, people, or points are interconnected.
\nWith this understanding of graphs, we can now explore how to navigate these connections to find the shortest paths using Breadth-First Search.
\nIn a previous article, we discussed binary search, an efficient algorithm for finding an item in a sorted array. Breadth-First Search (BFS) is another kind of algorithm entirely, one that leverages graphs to answer questions such as:
\nBFS operates on the principle of exploring the closest, or neighboring, vertices first before moving on to more distant ones. This approach ensures that if a path exists, it will be found as efficiently as possible. Let's understand BFS with an example involving a practical scenario.
\nImagine you own a mango farm and are looking for a mango seller who can market your harvest. You might start by asking, "Do I know any mango sellers on Facebook?" This search process is quite straightforward. First, make a list of friends to search through.
\n1. Juan\n2. Diogo\n3. Erick\n4. ...\nFor each friend, you check if they sell mangas:
\nSuppose none of your immediate friends are mango sellers. You would then need to look at your friends' friends.
\n![\"Mangoes](\"/_next/image?url=%2F_next%2Fstatic%2Fmedia%2Fmangoes-sellers.0fbce412.png&w=3840&q=75\"/)
\nAs you search each person on your list, if they aren't mango sellers, you add all their friends to your search list. This way, you're not only searching among your friends but also among their friends, and so on. This creates a search pattern that expands outwards, ensuring that you cover your entire network until you find a mango seller. This is Breadth-First Search at work.
\nWe've seen how to determine if a path exists between two vertices using BFS. Now, let's figure out how to find the shortest path between them. Can you find the closest mango seller? For instance, your direct friends are first-degree connections, while friends of friends are second-degree connections. Naturally, you'd prefer a first-degree connection over a second-degree one and so on.
\nIt's vital not to search through second-degree connections until you're sure there are no first-degree connections who are mango sellers. The way BFS works, it spreads out from the starting point, ensuring that you check all first-degree connections before moving on to second-degree ones. By following the order of the list—first-degree, then second-degree connections, and so on—you're guaranteed to find the closest mango seller. BFS doesn't just find any path; it finds the shortest path. This is only effective if you maintain the order in which connections are added to the list, adhering to their degree of separation.
\nIn the subsequent section, we'll discuss the role of queues in BFS and how they help maintain the order necessary to ensure the shortest path is found.
\nWhen dealing with the mechanics of the Breadth-First Search algorithm, the data structure that comes into play is the queue. A queue operates much like the queues we're familiar with in everyday life—think of a line of people waiting to buy tickets at a movie theater.
\nIn computer science, a queue is a collection where elements are added to the back and removed from the front, adhering to a First-In-First-Out (FIFO) policy. Imagine a lineup of people: the first person to join the line is the first person to be served and leave the line.
\nThe operations that can be performed on a queue are 'enqueue', which adds an element to the end of the queue, and 'dequeue', which removes an element from the front. If you enqueue two items, the first item will be dequeued before the second—just as in a real-world queue.
\nenqueue(item1) -> item1 is now at the back of the queue\nenqueue(item2) -> item2 is placed behind item1\ndequeue() -> item1 is removed from the front of the queue\nThis behavior is pivotal for the BFS algorithm as it ensures that vertices are explored in the order they were discovered. As you come across new vertices during your search, you enqueue them. When it's time to explore a vertex, you dequeue, ensuring you're always checking the oldest unexplored vertex first, which maintains the breadth-first nature of the search.
\nContrast this with a stack, a Last-In-First-Out (LIFO) structure where the last element added is the first to be removed, which is used in another type of graph traversal called Depth-First Search. The choice between using a queue or a stack is what distinguishes BFS from DFS, with each serving different types of problems and goals in graph exploration.
\nIn summary, queues in BFS allow the algorithm to keep track of which vertices to explore next and ensure that the search is conducted level by level, starting from the source vertex and moving outward, thereby guaranteeing the shortest path is found in an unweighted graph.
\nIn the realm of programming, the first step towards solving graph-related problems is to implement a graph in code. A graph comprises nodes, also known as vertices, which are interconnected through edges.
\nTo represent a relationship like "you → Bob", a helpful data structure is a hash table.
\nHere's a simple representation of a graph in JavaScript:
\nconst graph = {}\ngraph['you'] = ['alice', 'bob', 'claire']\nIn this graph, "you" is mapped to an array that contains all your neighbors. But what if we want to extend this graph to include more nodes and connections?
\nconst graph = {}\ngraph['you'] = ['alice', 'bob', 'claire']\ngraph['bob'] = ['anuj', 'peggy']\ngraph['alice'] = ['peggy']\ngraph['claire'] = ['thom', 'jonny']\ngraph['anuj'] = []\ngraph['peggy'] = []\ngraph['thom'] = []\ngraph['jonny'] = []\nIn this expanded graph, Anuj, Peggy, Thom, and Jonny don't have any neighbors. They have edges pointing towards them, but no edges from them to others. This type of graph is called a directed graph (or digraph), where the relationship holds true in only one direction.
\nFor a practical application of the Breadth-First Search algorithm, we'll find a mango seller within our network. Here's how we can achieve that:
\nHere's a JavaScript implementation of the above steps:
\n\nThe execution time of Breadth-First Search algorithm mainly involves checking every edge, implying a runtime of at least O(E), where E is the number of edges. In addition, a queue of people is maintained to be checked, with each insertion operation consuming constant time, O(1), leading to an overall time complexity of O(V) for all people, where V is the number of vertices.
Therefore, the total execution time is O(V + E), where V is the number of vertices (or nodes/people), and E is the number of edges (or connections). This efficient time complexity makes Breadth-First Search a useful choice for finding the shortest path in unweighted graphs.
Implementing and understanding graphs are key aspects of being a proficient software engineer. They are not only fundamental data structures, but their wide range of applications from social networks to geographical maps highlight their importance. The Breadth-First Search (BFS) that we covered in this article offers an efficient way to traverse or search a graph, commonly used for finding the shortest path in unweighted graphs. Remember its time complexity is O(V+E), making it efficient for large scale applications.
Though we used JavaScript in this example, knowledge of how to implement a graph and BFS algorithm transcends specific programming languages and is applicable across the board.
\nKeep honing your algorithms and data structures skills, as they significantly enhance your problem-solving capabilities as a software engineer.
\nFor more articles on latest technologies and guides on mastering foundational concepts, subscribe to the newsletter below.
\nKeep coding, and see you on the next one!\nimport from "react"
", "url": "https://jopcmelo.dev/articles/introduction-to-graphs", "title": "Graphs Unveiled: Exploring Shortest Paths with Breadth-First Search", "summary": "Delve into the implementation of graphs, understand the Breadth-First Search (BFS) algorithm, its application and time complexity. A handy guide for software engineers.", "date_modified": "2024-02-25T00:00:00.000Z", "author": { "name": "João Melo" } }, { "id": "https://jopcmelo.dev/articles/hash-tables", "content_html": "Let's take a journey into the world of a librarian. A reader wishes to read a particular book, say "War and Peace". You, as the librarian, scour your list of available books to find its reference number.
\nIf the list isn't alphabetically sorted, you could spend precious time going through every entry in an attempt to locate "War and Peace". This method closely resembles a linear search, an algorithm whose performance stands at O(n).
You may consider keeping the list alphabetically organized and then using a binary search to find the book. This drill could cut the searching time to O(log n), a significant improvement over the previous method.
However, making a reader wait even for these few moments is hardly an ideal scenario. What if we could instantly recall the reference number of any book on demand? Such a system with O(1) runtime would significantly enhance the user experience while reducing the waiting time to virtually zero.
To replicate this level of efficiency in our software projects, we turn to a powerful data structure - Hash Tables. This tool joins the line-up of array and linked list data structures, providing a powerful tool to fast-track data retrieval within your applications.
\nHash Functions allow us to build this highly efficient system. They are unique functions that take a sequence of bytes (like a string), and return a number; in essence, they map strings to numbers. Yet, for hash functions to work effectively in a Hash Table, they must follow certain crucial rules:
\nThe operation of a hash function becomes clear when visualized in steps:
\nCombine a hash function's unique mapping with an array, and voila! You have a powerful new data structure in your hands: a Hash Table. Hash Tables, hence, owe their efficiency to the intricate workings of hash functions.
\nHash Tables emerge as the first data structure that introduces a dose of logic in learning, marking a departure from the direct mapping to memory that we see in arrays and linked lists. Hash Tables are indeed smarter and more efficient.
\nHow is that? They ingeniously leverage a hash function to decide precisely where to store elements. As a result, Hash Tables are among the most useful, albeit complex, data structures that you will learn.
\nThese clever constructs are often known as hash maps, maps, dictionaries, and dispersion tables, hinting at their extensive utilization. One of the key advantages of Hash Tables is their high speed and efficiency, which comes in handy in many applications.
\nImportantly, you'll seldom need to implement a Hash Table from scratch. Almost every programming language will have one pre-built for you. Here's what a Hash Table looks like in JavaScript:
\nconst hash = {}\nhash['War and Peace'] = 4578\nhash['1984'] = 1532\nconsole.log(hash['War and Peace']) // 4578\nObserve how each hash table contains a key (book title in this case) and a corresponding value (reference number). It's their unique amalgamation within a Hash Table that paves the way for their on-demand and instant retrieval.
\nHash Tables are versatile and can be applied in various scenarios due to their effective data retrieval capabilities. Let's delve into some practical applications.
\nConsider your school with various classrooms, each assigned a unique code. Suppose you want to create a system to map classroom codes to the teachers assigned to them. This system should:
\nThis scenario is the perfect use case for Hash Tables, which are ideal when mapping from one item to another or performing a lookup. Let's use a Hash Table for this:
\nconst classroomMap = {}\n\n// Add a teacher's name and a classroom code\nclassroomMap['Donald E. Knuth'] = 'Room A123'\nclassroomMap['Alan Kay'] = 'Room B456'\n\n// Find the classroom code\nconsole.log(classroomMap['Donald E. Knuth']) // Room A123\nImagine you're managing entries for an event where each guest may only enter once. How do you check if a guest has already entered? You could maintain a list of guests that have already entered and check their full name against it. But this method gets cumbersome when dealing with large crowds and extensive lists.
\nInstead, we could utilize a Hash Table!
\nconst enteredGuests = {}\n\n// Check if guest already entered\nif (enteredGuests['Erick Wendel']) {\n console.log('Already entered!')\n} else {\n enteredGuests['Erick Wendel'] = true\n console.log('Welcome!')\n}\nWith this Hash Table, we can instantly check if a guest has already entered, saving precious time and computing resources.
\nWeb developers often endorse caching as a best practice. When you visit websites like Amazon.com, the server may take a few seconds to gather all the featured deals and top-selling products. These few seconds can seem like an eternity to users who might perceive Amazon to be sluggish.
\nCaching plays a crucial role here, acting like a memory bank that recalls data instead of recalculating it every time. When you revisit Amazon, the server checks if the home page is stored in its cache (which we can represent with a Hash Table). If it finds a cached version, it quickly serves out the webpage.
\nThis caching technique has two primary benefits:
\nHere's a simplified pseudo-code representation of such caching:
\nlet cache = {}\n\nasync function getCachedPage(url) {\n if (cache[url]) {\n return cache[url]\n } else {\n const page = await fetchFromServer(url)\n cache[url] = page\n return page\n }\n}\nHash collisions are a common challenge that programmers face when working with hash tables. So, what exactly is a hash collision, and how do we deal with it?
\nCollisions happen when two keys map to the same slot in a hash table. Although hash functions aim to provide a unique mapping, achieving a perfect hash function, which guarantees that each key will map to a different slot, isn't always feasible.
\nLet us consider a scenario where we keep track of the population of American cities. Suppose we have an array with 50 slots, each corresponding to a U.S. state, and the hash function assigns each city to an array slot based on which state the city belongs to. All's well initially. The population of Augusta (Maine) is stored in the first slot, and Annapolis (Maryland) goes into the second. But things get problematic when we try adding Auburn (Alabama). Our hash function maps it to the first slot, which is already holding Augusta's data. Hence, a collision has occurred.
\nThis collision is problematic, as if we store Auburn's population in this slot, it will overwrite Augusta's population. So, when we want to retrieve Augusta's population, the hash table will incorrectly give us Auburn's population.
\nOne way to handle collisions is by using a linked list. Whenever multiple keys map to the same slot, you simply start a linked list at that slot.
\nThis solution works fine if your linked list is relatively small. However, imagine having a linked list with thousands of city data entries. Searching through it would be considerably slow.
\nFrom this, we learn two important lessons:
\nIn short, preventing collisions is all about designing a robust hash function that mitigates the risk of collisions and ensures uniform key distribution.
\nAmong the various data structures available, hash tables are acclaimed for their excellent performance. To understand why, let's delve into how their efficiency is quantified and how to ensure optimal performance.
\nIn terms of time efficiency, hash tables are exceptional. In the average case, all operations – search, insert, and delete – are executed in O(1) time. This is referred to as constant time. Constant time doesn't mean the operation is instant, but rather that the time required remains the same regardless of the hash table's size.
However, the worst-case scenario tells a different story. If things go south, the time efficiency becomes O(n), i.e., linear time, for all operations. This is much slower and usually due to collisions.
Let's look at a comparative analysis of hash tables, arrays, and linked lists:
\n| Hash tables (Average case) | Hash tables (Worst case) | Arrays | Linked lists | |
|---|---|---|---|---|
| Search | O(1) | O(n) | O(1) | O(n) |
| Insert | O(1) | O(n) | O(n) | O(1) |
| Delete | O(1) | O(n) | O(n) | O(1) |
In the average case, hash tables excel by rivaling arrays in search time and linked lists in insert/delete times - it’s the best of both worlds. But the worst-case scenario reveals a direr performance. Therefore, it's crucial to avoid worst-case scenarios by preventing collisions. How? Through a low load factor and a good hash function.
\nThe load factor of a hash table indicates how filled the hash table is. It’s calculated as the number of items in the hash table divided by the total number of slots. For example, a load factor of 0.5 means that half of the hash table's slots are filled, while a load factor of >1 indicates overfilling.
\nManaging the load factor is vital to the performance of the hash table. As the load factor rises, the likelihood of collisions increases, slowing down the hash table’s performance. A widely accepted threshold is to resize the hash table when the load factor exceeds 0.7.
\nResizing is the process of increasing the hash table's capacity when its load factor breaches an upper limit. Essentially, you're making more room for new entries and reducing the collision probability.
\nTo resize, start by doubling the existing array size. Then, you'll have to rehash all the items using the hash function and insert them into this newly expanded hash table. With this increased capacity, the load factor drops, thereby decreasing chances for collisions and improving the performance of your hash table.
\nUnderstanding hash tables, collisions, and maintaining high performance are essential for every software engineer handling large datasets. Hash tables emerge as one of the most efficient data structures due to their constant time complexity across search, insert, and delete operations. However, their effectiveness significantly depends on minimizing collisions and appropriately maintaining the hash tables' load factor.
\nQuality hash functions contribute to evenly distributing data across the table and reducing collisions, but they aren't always sufficient. Once we encounter more collisions, and our load factor exceeds certain thresholds (0.7 is commonly used), it's essential to resize our tables, creating a larger space for data and thereby reducing the probability of future collisions.
\nWhile hash tables can't always guarantee constant time complexity, with a carefully constructed hash function and an attentive eye on load factors, you can ensure you’re close to meeting that benchmark.
\nIn essence, hash tables are highly desirable for their speedy operation times, which are achievable under ideal conditions. Yet, understanding the potential for hash tables to operate at linear time efficiency is equally important. Hence, monitoring and optimizing the load factor are crucial practices in working with hash tables.
\nIf you enjoyed this deep-dive into hash tables and want to keep the ball rolling on your data structure journey, don't forget to subscribe to my newsletter.
", "url": "https://jopcmelo.dev/articles/hash-tables", "title": "Mastering Hash Tables: From Understanding to Optimization", "summary": "A comprehensive look at hash tables and practical tips to optimize their utilization. Dive into hash functions, load factors, and trade-offs to learn how to get the best of hash tables.", "date_modified": "2023-11-28T00:00:00.000Z", "author": { "name": "João Melo" } }, { "id": "https://jopcmelo.dev/articles/quicksort", "content_html": "Quicksort, a renowned sorting algorithm boasting its high speed compared to selection sort, frequently wins the hearts of programmers around the world. This article aims to illuminate its methodology and nuances using straightforward examples that delve into the handling of arrays with various lengths.
\nStraightforward as it may seem, a methodical understanding of quicksort begins with the most rudimentary array - an empty array or an array with a single element. In sorting such an array—voila—it's already sorted! Thus, these cases form our base case.
\nMoving on, let's consider an array with two elements. The algorithm's job is to ensure that the first element is smaller than the second. Otherwise, it flips them.
\nThings get trickier once we escalate to a three-element array. It's important to remember that quicksort is a faithful practitioner of the Divide and Conquer strategy.
\nThe aim is to decompose the array until we reach our base case. Consequently, in quicksort, we assign an anchor role to one element of the array, called the pivot. For simplicity, initially, we elect the first item of the array as our pivot.
\nThis leads us to perform partitioning, where we identify elements smaller than the pivot and those greater. Post-partitioning, you'll have:
\nDo note that both these subarrays are partitioned but not sorted. How do we sort them then? Bob's your uncle here, as our base case has already paved the way. Apply quicksort recursively to both subarrays and combine the results.
\nOnce you've mastered sorting a three-element array, you can tackle an array with more elements using the same approach. Let's consider a four-element array where we choose the largest number as the pivot. Consequently, one of the partitions will contain the remaining three elements, which we already know how to handle!
\nOnce you grasp this pattern, sorting larger arrays becomes a breeze. For an illustrative JavaScript implementation of Quicksort, visit our GitHub repository or see the code snippet below.
\nfunction quicksort(array) {\n if (array.length < 2) {\n return array\n }\n\n const pivotIndex = Math.floor(Math.random() * array.length)\n const pivot = array[pivotIndex]\n array.splice(pivotIndex, 1)\n\n const lower = array.filter((el) => el <= pivot)\n const higher = array.filter((el) => el > pivot)\n\n return quicksort(lower).concat([pivot], quicksort(higher))\n}\n\nconsole.log(quicksort([10, 5, 2, 3])) // [2, 3, 5, 10]\nSelecting the pivot in Quicksort is not a trivial task. In fact, the pivot choice significantly influences the efficiency of the entire sorting process.
\nTo understand this, imagine a scenario where you always pick the first element as the pivot. Now consider what happens when quicksort is applied to an already sorted array. Interestingly, Quicksort does not have a mechanism to check if an array is pre-sorted, which means it proceeds to attempt sorting it anyway.
\nThe issue here is that the sorting process does not yield two balanced partitions due to the specific pivot choice (the first element). In other words, you are not truly dividing your problem into smaller, equal parts. What you get instead is an entirely empty subarray and another that has to absorb all the elements. This imbalance subsequently leads to excessive recursion, flooding the call stack, which induces performance inefficiencies.
\nOn the contrary, opting to choose the middle element as the pivot proffers more balanced partitions. The idea here is to distribute the sorting load more equally between two smaller subarrays. This approach effectively splits the problem into two approximately equal parts, minimizing the need for excessive recursive executions. Evidently, each separate sorting task is easier and quicker to handle, thereby contributing to a more efficient Quicksort process overall.
\nIn other words, a simple choice in pivot selection can drastically alter how the stack call is utilized, thereby potentially augmenting or hindering the efficiency of the sorting. Therefore, choosing a random element from the array as the pivot whenever Quicksort is used emerges as an efficient strategy to enhance the overall performance and limit the recursions.
\nMastering quicksort requires understanding its base cases, implementation of the Divide and Conquer strategy, and the pivotal role of the pivot. Just like a well-sorted array, understanding these building blocks will empower you to sort any array, no matter its size.
\nAnd remember, choice matters—especially when choosing the pivot!
\nDid this article help you better understand quicksort? The journey doesn't end here. Subscribe to my newsletter below and master more concepts!
", "url": "https://jopcmelo.dev/articles/quicksort", "title": "Mastering Quicksort: A comprehensive Guide for Developers", "summary": "Unravel the workings of the Quicksort algorithm, right from its application on simple arrays to handling larger ones. Discover how the 'Divide and Conquer' strategy plays out in Quicksort, and learn the underestimated significance of pivot choice in boosting efficiency.", "date_modified": "2023-11-26T00:00:00.000Z", "author": { "name": "João Melo" } }, { "id": "https://jopcmelo.dev/articles/divide-and-conquer", "content_html": "Have you ever faced a daunting task, like a jigsaw puzzle, and wondered where to start? Most of us would instinctively begin by sorting the pieces into small manageable groups, maybe by color or edge type. This illustrates a common problem-solving strategy, breaking a problem down into smaller parts to make it more manageable. In computer science, we use the same principle with a strategy known as "Divide and Conquer". In this article, we dive into the fundamentals of "Divide and Conquer" and show how it can be applied with recursive algorithms, using real-world examples.
\n![\"Jigsaw](\"/_next/image?url=%2F_next%2Fstatic%2Fmedia%2Fjigsaw-puzzle.1bb23e68.png&w=2048&q=75\"/)
\n"Divide and Conquer" is a popular problem-solving strategy in the field of computer science. It involves breaking down a complex problem into smaller subproblems. These subproblems are easier to solve, and their solutions can then be combined to address the original issue. Essentially, you're turning a big, complex problem into a series of more manageable tasks.
\nThe key steps of the divide and conquer technique are:
\nThis technique forms the cornerstone of recursive algorithms, an important computer science concept I discussed in a previous article. Let's start applying these principles to our first problem.
\nTo bring this concept to life, let's consider a farmer who wishes to divide a plot of land, measuring 1680m by 640m, into equal square portions. The challenge here is to determine the maximum size of these square parcels.
\nFollowing the Divide and Conquer approach, our first step is to identify the base case. For this scenario, a simple base case would be if one of the sides were a multiple of the other. For instance, if we had a 50m by 25m plot, we could easily divide it into two squares each measuring 25m by 25m.
\n![\"Farm](\"/_next/image?url=%2F_next%2Fstatic%2Fmedia%2Ffarm.27f73162.png&w=3840&q=75\"/)
\nNow on to the next step: dividing the original problem until we encounter our base case. This is the point where Divide and Conquer really shines. Starting with the vast land area, we first identify the largest square that fits: in this case, a 640m x 640m square. We fit as many of these squares as possible into our area of land and then look at what's left over.
\nFollowing this process, we find that two 640m x 640m squares can fit within our plot, leaving behind a 400m x 640m area. From this point, we now need to divide this residual area in the same approach as before, shrinking our problem at every turn. The largest square that could fit in this remaining area is 400m x 400m which leaves a 240m x 400m area undivided.
\nContinuing this strategy, we fit in another square, this time 240m x 240m, leaving a 160m x 240m area. We then add a 160m square and are left with an area of 160m x 80m. At this point, we have reached our base case, as 160 is a multiple of 80. Dividing this last segment into squares leaves no segments unused. Therefore, for the original farmland, the largest square unit we can use to divide it into equal squares is 80m x 80m.
\nThis technique also has many practical applications in programming tasks. For instance, if you want to sum all the numbers in an array, you could easily execute this with a for loop:
\nfunction sum(arr) {\n let total = 0\n for (let i = 0; i < arr.length; i++) {\n total += arr[i]\n }\n return total\n}\nHowever, this problem could also be solved recursively, using the Divide and Conquer technique. By defining our base case as an array with length 0 (or 1), we continuously reduce the size of the array with each recursive call until we reach the base case:
\nfunction sum(arr) {\n if (arr.length === 0) {\n // returns the base case\n return 0\n }\n return arr[0] + sum(arr.slice(1)) // returns the sum of the first element and the sum of the rest of the array\n}\nNotice how the function reduces the problem size, brings it closer to the base case with each recursive call, and finally halts when it hits the base case. That is Divide and Conquer in action!
\nFurther practice with the Divide and Conquer technique helps cement your understanding of recursive functions. Try your hand at the following exercises.
\nfunction count(arr) {\n if (arr.length === 0) {\n // returns the base case\n return 0\n }\n return 1 + count(arr.slice(1)) // returns 1 plus the count of the rest of the array\n}\nfunction max(arr) {\n if (arr.length === 1) {\n // returns the base case\n return arr[0]\n }\n return Math.max(arr[0], max(arr.slice(1))) // returns the highest value between the first element and the highest value of the rest of the array\n}\nIn summary, the Divide and Conquer strategy allows us to tackle complex problems by breaking them down into more manageable subproblems. This technique then employs recursion to systematically solve these problems. Whether you're simplifying a land division problem or programming an array summing algorithm, this strategy greatly facilitates organized problem-solving.
\nMost importantly, this technique forms the foundation of various efficient sorting algorithms such as Quick Sort and Merge Sort, demonstrating its versatility and its critical role in computer science. Now that you've learned the fundamentals of "Divide and Conquer" techniques and their implementation in recursive algorithms, take some time to practice and solidify your understanding.
\nAs with any skill, mastering these techniques comes with consistent practice. To guide your practice similar problems, I've included the code from this article and additional related examples in an open-source GitHub repository. Feel free to explore it at this link.
\nIf you enjoyed this article and want even more great content like this, subscribe to my newsletter. Feel free to explore previous articles on recursion, selection sort, and other insightful subjects. Thanks for reading!
", "url": "https://jopcmelo.dev/articles/divide-and-conquer", "title": "Deconstructing Problems with \"Divide and Conquer\" Technique", "summary": "Unlock the power of \"Divide and Conquer\" strategy in problem-solving. Learn how to solve complex problems with simplicity using recursive algorithms. Master this fundamental tool of computer science, and apply it in real-world examples from farmland division to summing elements in an array.", "date_modified": "2023-11-10T00:00:00.000Z", "author": { "name": "João Melo" } }, { "id": "https://jopcmelo.dev/articles/recursion", "content_html": "Think about a Russian doll. You open the first doll to find a smaller doll inside, then another one in that, and this continues until you can't find any more dolls. This real-world scenario exhibits recursion, where a big problem (finding the smallest doll) can be solved by dividing it into smaller, simpler problems of the same type.
\nRecursion is pervasive in computer programming, enabling developers to tackle complex chaos with simple elegance. However, to some, it can appear daunting, or even unnecessary – when can't I just use a loop? I aim to dispel this unease by lending you my understanding of recursion, built especially for software engineering, with all of its beauty, drawbacks and common applications.
\n![\"A](\"/_next/image?url=%2F_next%2Fstatic%2Fmedia%2Frussian-doll.6b72e123.png&w=2048&q=75\"/)
\nThe suitcase conundrum: imagine you're rummaging through your basement and stumble upon a mysterious, locked suitcase. You suspect the key is hiding in one of many boxes lying around, but these boxes can further contain smaller boxes. A recursive solution to finding the key looks like this:
\nOn the other hand, you could set up a loop that throws all boxes on a pile, reviewing each until the key is found. While these strategies hit the same outcome, recursion undoubtedly adds an intuitive clarity to the answer's pathway.
\nInterestingly, this elegance does not translate into performance benefits. Sometimes, loops even have the upper hand when considering system resources.
\nA recursive function needs to know when to stop calling itself, ensuring we're safe from a dreaded infinite loop. Therefore, every recursive function has two essential parts:
\nThe 'stack' concept is integral to understanding recursion - and can be easily imagined by considering a pile of sticky notes. Each note could represent a task for, let's say, an upcoming BBQ party. You add tasks on top of the stack (push), and you deal with them from the top too, reading and removing as you go (pop) – that's a 'Stack'.
However, recursion deals with a specific version - 'Call Stack'. Every time a function is activated, a new frame is pushed onto the Call Stack. This Stack keeps track of the function to return to post the current function's completion. The concept extends to recursive functions, with every recursive call adding a new frame. The base case comes to rescue when all function calls have completed executing, popping frames off the Call Stack until it's empty.
\nLet's dissect this on a classic example: factorial function.
\nfunction factorial(x) {\n if (x === 1) return 1\n\n return x * factorial(x - 1)\n}\nThe factorial function calls itself to find the product of a number and the factorial of the one before it. When x equals 1, it stops the recursive calls, popping the frames and returning the computed value.
While the Call Stack aids recursion, overusing it might lead to memory issues. If you're saving numerous calls in memory (you've a very long stack), you could either rewrite your function to use loops instead of recursion or use tail recursion - a complex concept beyond the scope of this post, with limited support across programming languages.
\nA practical example of recursion is counting the total number of files in a directory and its sub-directories.
\nfunction countFiles(directory) {\n let count = directory.files.length\n\n for (const subDirectory of directory.subDirectories) {\n count += countFiles(subDirectory)\n }\n\n return count\n}\nArriving at simpler problems, consider summing an array of numbers through recursion.
\nfunction sum(array) {\n if (array.length === 0) return 0\n\n return array[0] + sum(array.slice(1))\n}\nComputing the Fibonacci series is a classic recursion job - each number equals the sum of the two preceding ones.
\nfunction fibonacci(n) {\n if (n <= 2) return 1\n\n return fibonacci(n - 1) + fibonacci(n - 2)\n}\n\nconsole.log(fibonacci(7)) // Output: 13\nSo we've deciphered recursion. Gaining comfort with recursion can take a little time. Breaking up complex problems into smaller, similar problems - over and over again until a solution surfaces should become more natural with practice. It's a powerful tool that can effectively clear a lot of conceptual clutter, lend elegance to your solution and breed efficient code when used appropriately. So gear up for some recursive coding!
", "url": "https://jopcmelo.dev/articles/recursion", "title": "Recursion Revealed: An Intuitive Guide for Software Engineers", "summary": "Delve into the depths of recursion, an elegant and mighty concept in coding. With an array of real-life examples and code snippets, we'll unravel the 'mystery' of recursion, proving that it's not as daunting as it seems! Step in and immerse yourself into the world where problems solve themselves.", "date_modified": "2023-11-06T00:00:00.000Z", "author": { "name": "João Melo" } }, { "id": "https://jopcmelo.dev/articles/selection-sort", "content_html": "So you've dipped your toes with arrays and linked lists (here's a link to the tutorial for a quick refresher), and you've understood the Big O Notation (check out the article if you haven't yet). It's time to unpack the mechanics of the Selection Sort algorithm.
\n![\"Selection](\"/_next/image?url=%2F_next%2Fstatic%2Fmedia%2Fselection-sort.8110097d.png&w=2048&q=75\"/)
\nLet's imagine you have a list of artists based on their play-count on your computer:
\n| Artist | Plays count |
|---|---|
| Van Halen | 132 |
| The Rolling Stones | 353 |
| Metallica | 220 |
| Nirvana | 185 |
| Aerosmith | 157 |
| Led Zeppelin | 238 |
| Red Hot Chili Peppers | 201 |
You'd like to sort this list from high to low play-count to categorize your favorite artists. One way to do it would be to pull the most played artist from your list and add it to a new one, then repeat this process until everybody's accounted for:
\n| Artist | Plays count |
|---|---|
| The Rolling Stones | 353 |
| Led Zeppelin | 238 |
| Metallica | 220 |
| Red Hot Chili Peppers | 201 |
| Nirvana | 185 |
| Aerosmith | 157 |
| Van Halen | 132 |
To find the artist with the most plays, we check each item in the list, a process with run-time O(n). To find the second most popular artist, you'll again need to check the list item by item, again O(n). As you see, when you want to find the next most popular artist, you run the same code n times. This makes the running time O(n²).
You'd be right to observe that as you proceed, the number of elements to examine shrinks, eventually leaving only a single element. So why the O(n²) complexity? The answer lies in the specifics of Big O Notation.
You first check n elements, then n - 1, n - 2, and so forth, which averages out to checking 1/2 * n elements. The running time becomes O(n * 1/2 * n). However, in Big O Notation, we disregard constants like 1/2, leaving us with O(n * n) or O(n²).
Although a good algorithm, the Selection Sort isn't particularly fast. The Quicksort, for example, is faster with a complexity of O(n log n)— a topic for another post.
Here's a code snippet of how you'd sort an array of numbers using the selection sort algorithm in JavaScript:
\nfunction searchForSmallest(arr) {\n let smallest = arr[0]\n let smallestIndex = 0\n\n for (let i = 0; i < arr.length; i++) {\n if (arr[i] < smallest) {\n smallest = arr[i]\n smallestIndex = i\n }\n }\n\n return smallestIndex\n}\n\nfunction selectionSort(arr) {\n const newArr = []\n\n while (arr.length) {\n const smallestIndex = searchForSmallest(arr)\n newArr.push(arr.splice(smallestIndex, 1)[0])\n }\n\n return newArr\n}\n\nconsole.log(selectionSort([5, 3, 6, 2, 10])) // [2, 3, 5, 6, 10]\nYou can find this script in my GitHub algorithms repository here. A more visual and interactive dig into the algorithm is available in this Nodepad note I've prepared for you.
\nAnd there you have it, folks! The selection sort algorithm might not be the fastest, but it's straightforward and a great starting point for understanding the mechanics of sorting algorithms. It thrives in simplicity, and its concept can be translated easily into real-world tasks.
\nRemember, in programming (and in life), understanding the basics is key to tackling even the most complex problems. As you've ventured into Selection Sort, remember that this foundation will carry you forward in your journey towards comprehending more efficient algorithms.
\nKeep your eyes on our future discussions about sorting algorithms like Quicksort, and let's continue expanding our arsenal together!
\nDon't hesitate to revisit this piece anytime you need to refresh your Selection Sort knowledge. As always, happy coding!
", "url": "https://jopcmelo.dev/articles/selection-sort", "title": "Demystifying Sorting Algorithms: A Deep Dive into Selection Sort", "summary": "Break into the world of algorithms with an in-depth look at Selection Sort! With practical examples and easy-to-follow JavaScript code, this comprehensive guide will turn you into a sorting master. No fluff, just wisdom!", "date_modified": "2023-10-29T00:00:00.000Z", "author": { "name": "João Melo" } }, { "id": "https://jopcmelo.dev/articles/arrays-and-linked-lists", "content_html": "In the programming universe, two core data structures will frequently cross your path: arrays and linked lists. They have distinct characteristics, benefits, and cases where one might outshine the other. We've previously explored arrays when we delved into binary search. If you need a refresh, don't hesitate to revisit our binary search article.
\nBefore we dive into arrays and linked lists, make sure you're comfortable with Big O notation--it will greatly simplify the performance analysis in this tutorial. You can brush up on it here.
\nThink of computer memory as an expanse of gym lockers, each with its unique address. Want to store something? You get a locker. Need to store more? You'll need more lockers.
\nSimilarly, when you save an item in computer memory, the computer assigns it an address. For multiple items, you'll utilize arrays or linked lists.
\n![\"Explaining](\"/_next/image?url=%2F_next%2Fstatic%2Fmedia%2Funderstanding-computer-memory.97f4c3dc.png&w=2048&q=75\"/)
\nTo illustrate, let’s suppose you're building a grocery shopping app. Storing items requires either an array or a linked list. The former stores items side by side in memory.
\nHowever, arrays can pose challenges. Suppose you need to add another item, but the next memory slot is occupied. You'll need to find a larger memory region, move all items, and add the new one. Quite the hassle, right?
\nA possible solution is to reserve more space than required, offering room for additions. But that approach wastes memory space if unused, and if the space fills up, you're back at square one.
\nLinked Lists offer a different approach. They allow your items to be located anywhere in memory, with each item pointing to the next one. Similar to a digital treasure hunt, you hop from one address to the next. This means you can add items anywhere in memory without relocating existing ones.
\n![\"Understanding](\"/_next/image?url=%2F_next%2Fstatic%2Fmedia%2Flinked-lists.1dcaa07f.png&w=2048&q=75\"/)
\nYou know how "top 10" websites have a sneaky tactic for getting more views? Instead of displaying the full list on a single page, they place one item per page and make you click "next" to view the subsequent item. It's annoying, but it earns them more from ad revenue.
\nIt would be considerably more user-friendly if the entire list were on one page, and you could click on each item's name for more details.
\nLinked lists have a similar pitfall. Suppose you want to read the last item of a linked list. You can't do that directly as you don't know its address. Instead, you have to go to item #1 to fetch the address for item #2, then move to item #2 to get the address for item #3, and so on until you reach the last item.
\nLinked lists work perfectly fine if you want to read all items, one by one. However, if you want to jump from one item to another, linked lists score low on convenience.
\nArrays, on the other hand, operate differently. If you want to read the last item of an array, you can directly go to the address of the last item. No need to read all the items preceding it.
\nIn an array, elements are assigned an index. This indexing begins at 0, not 1. Take this array, for instance, the number 20 is at index 1.
\n[10, 20, 30, 40]\nThe number 10 is at index 0. This might confuse people initially, but that's how it works.
\nThe position within an array is referred to as the index. The index of the number 20 is 1. The index of the number 10 is 0. The index of the number 40 is 3.
\nHere's how arrays and linked lists compare for common operations:
\n| Arrays | Linked Lists | |
|---|---|---|
| Reading | O(1) | O(n) |
| Insertion | O(n) | O(1) |
Imagine you want your grocery list to be alphabetically ordered. That means now, you'll want to add items in the middle of the list - simply appending the item at the end of the list might violate the alphabetical order.
\nWhat's the better choice for inserting elements in the middle of a list: arrays or linked lists? Using linked lists, you just need to adjust what address the preceding item points to.
\nFor arrays, however, you must shift all the items below the insertion address. If there isn't enough room, you'll have to request more space and move every item over there.
\nIn essence, linked lists have the upper hand when it comes down to inserting items in the middle of the list.
\n![\"Grocery](\"/_next/image?url=%2F_next%2Fstatic%2Fmedia%2Fgrocery-list.5ef9bfbb.png&w=2048&q=75\"/)
\nWhat if you want to delete an item? Once again, linked lists make this easier as you just need to adjust what address the preceding item points to.
\nWith arrays, everything needs to be shifted when an element gets eliminated. Unlike insertions, deleting elements will always be successful. Insertion may fail when there isn't enough memory space.
\nHere are the runtimes for the most common operations for arrays and linked lists:
\n| Arrays | Linked Lists | |
|---|---|---|
| Reading | O(1) | O(n) |
| Insertion | O(n) | O(1) |
| Deletion | O(n) | O(1) |
Which one is more frequently used - arrays or linked lists? Clearly, it hinges on the scenario they're used in. However, arrays are more commonplace since they allow random access. There are two types of access: sequential and random.
\nSequential access means reading elements one by one, starting with the first.\nLinked lists can only manage sequential access. If you want to read the tenth element\nof a linked list, you first need to read the preceding nine elements to get the tenth\nelement's address.
\nRandom access, on the other hand, lets you jump straight to the tenth element.
\nRandom access is much needed in many cases, leading to wider usage of arrays.\nArrays and linked lists are employed to build other data structures, such as queues,\nstacks, trees, and graphs.
", "url": "https://jopcmelo.dev/articles/arrays-and-linked-lists", "title": "Arrays vs Linked Lists: An In-depth Comparison and Understanding", "summary": "Dive into this comprehensive guide discussing arrays and linked lists: two basic data structure types. Explore their distinction in detail, performance analysis using Big O Notation, and how to leverage their strengths based on your specific application needs.", "date_modified": "2023-10-27T00:00:00.000Z", "author": { "name": "João Melo" } }, { "id": "https://jopcmelo.dev/articles/big-o-notation", "content_html": "When writing or utilizing algorithms in real-life applications, we need to understand their performance characteristics - how fast or slow they can be. This is especially relevant when algorithms become part of specialized and critical systems like the code that determines the landing spot of a SpaceX rocket preparing to land on Mars.
\nThink about a developer named Fred who is in charge of that code. He has to choose between two different algorithms - a simple search or binary search.
\nBoth algorithms serve the same purpose, but they grow at different rates. Binary search is faster, a desirable attribute given that Fred has merely 10 seconds to calculate the landing spot. On the other hand, implementing simple search is much easier, representing a lower risk of bugs - rather crucial because nobody wants a bug when landing a spaceship!
\n![\"Big](\"/_next/image?url=%2F_next%2Fstatic%2Fmedia%2Fbig-o-notation.8e4cd26d.png&w=2048&q=75\"/)
\nTo make a wise decision, Fred decides to measure both algorithms against a list of 100 elements. He assumes that each element that he searches in the list will take 1ms.
\nThe simple search algorithm will need 100ms (because it takes O(n) time, where n is the number of elements), while the binary search algorithm will only need 7ms (given its O(log n) time complexity). But what happens if the size of the list increases significantly? To put it into perspective, if the list had around 1 billion elements, the binary search would take ~30ms, while a simple search would take around 11 (yes, eleven) days.
This hypothetical scenario showcases how the execution time of different algorithms can multiply at varying rates, demonstrating the importance of time complexity analysis. It's not about only the time an algorithm takes, but how it scales with input size. And that's where Big O Notation comes into play.
\n| # of elements | Simple Search | Binary Search |
|---|---|---|
| 100 | 100ms | 7ms |
| 10,000 | 10 seconds | 14ms |
| 1,000,000,000 | 11 days | 32ms |
Big O Notation is a mathematical notation that represents the upper bound - the worst-case scenario - in terms of time complexity for an algorithm. To take a few examples:
\nO(log n), also known as Logarithmic Time.O(n), also known as Linear Time.O(n * log n).O(n²), referred to as Quadratic Time.O(n!), or Factorial Time.Let's dive deeper into the Traveling Salesman Problem, as it's a famous example of an algorithm with frighteningly high time complexity.
\nThe Traveling Salesman Problem is a combinational optimization problem. It concerns a salesman who needs to travel to multiple cities, visit each one just once, and return to the original point. The aim is to find the shortest possible route.
\n![\"Traveling](\"/_next/image?url=%2F_next%2Fstatic%2Fmedia%2Ftraveling-salesman-problem.7d0d5d13.png&w=2048&q=75\"/)
\nHere you can see how the growth of this algorithm scales factorially with the number of cities:
\n| Cities | Operations |
|---|---|
| 6 | 720 |
| 7 | 5,040 |
| 8 | 40,320 |
| ... | ... |
| 15 | 1,307,674,368,000 |
| ... | ... |
| 30 | 265,252,859,812,191,058,636,308,480,000,000 |
In summary, for n items, it takes n! (factorial of n) operations to reach a result. So, that's O(n!) time complexity or factorial time.
Time complexity analysis and Big O Notation are crucial for designing efficient algorithms and systems. Whether writing a function to sort a small list or coding rocket landing software, understanding how your program's running time scales with the size of the input might make the difference between code that runs in milliseconds and code that runs for days.
\nSpace complexity is a term used in the study of algorithms to describe the amount of memory (space) an algorithm requires to execute. Understanding space complexity is as important as knowing time complexity, as it helps you consider the efficiency of your algorithm from a different angle.
\nOften, a trade-off exists between time and space complexity. Sometimes, by using more space (i.e., storing pre-computed values), you can decrease the time complexity of your algorithm (i.e., by reusing these stored values instead of re-calculating them). This is commonly referred to as a space-time tradeoff.
\nLet's break this down through an example related to social networks. Suppose you construct an algorithm to find whether two users are friends.
\nOne approach would be to run through all connections of user A, and for each connection, check if it’s user B (simple linear search). This approach involves low space complexity (you're not storing much extra data), but the time complexity is O(n), where n is the number of friends of user A.
\nAnother approach would be to utilize a data structure known as a hash set for storing all friends of user A. So, when you need to check if user B is a friend of A, you can look it up in the set directly. This way, the time complexity improves to O(1), but you've increased the space complexity because you store extra data in the hash set.
\nIn the real world, understanding Big O notation has a massive influence on how solutions are developed for large-scale systems. Here are a few examples:
\nIn search engines like Google, data structures known as inverted indices are used to store and search for information. These indices use a sorted list of keywords, each associated with a list of documents (webpages) containing the keyword. Searching for web pages related to a keyword thus becomes a matter of searching for the keyword in the index – a task well-suited for binary search, which has a performance of O(log n). This is critical when dealing with the enormous number of keywords and web pages indexed by a search engine.
\nBig O Notation plays a significant role in how services like Netflix make recommendations. They use algorithms that must be able to quickly analyze large amounts of data (i.e., previously watched movies, viewer ratings, and preferences) and make recommendations, requiring algorithms that scale well with an increase in data size. A poorly designed algorithm (with, say, quadratic time complexity) could lead to slow recommendations, negatively impacting user experience.
\nDatabases use various algorithms and data structures to store and retrieve data efficiently, and Big O notation describes the performance of these operations. For instance, efficient databases make use of B-Trees for indexing data due to its logarithmic time complexity which ensures that operations like insertion, deletion and search can be performed rapidly even on large datasets.
\nUnderstanding Big O notation ranks among the key steps in becoming a competent developer. Whether you're writing simple code to sort a list, or working on complex, large-scale applications dealing with billions of data points, an awareness of time and space complexity helps you choose the right algorithms, data structures, and design decisions.
\nBig O Notation provides a lens through which you can see potential bottlenecks in your programs and empower yourself to write code that runs efficiently, irrespective of your application's scale. It helps you comprehend the trade-offs between time and space and optimize according to the specific needs of your project.
\nIn real-world applications ranging from search engines to streaming services and database design, we see the power of a grasp on Big O Notation: faster, more efficient systems that can handle increasing loads of data.
\nHappy coding, and remember - the quest for efficiency is a developer's journey with no end, just like the unending learning and growth we get to experience in this fascinating field.
\nUntil next time, keep exploring, keep coding, and most importantly, keep optimizing!
", "url": "https://jopcmelo.dev/articles/big-o-notation", "title": "Grasping Big O Notation: Your Algorithm's Speedometer", "summary": "Big O Notation is a special form of notation that helps us understand the performance characteristics of an algorithm - how fast the algorithm grows in terms of time complexity.", "date_modified": "2023-10-23T00:00:00.000Z", "author": { "name": "João Melo" } }, { "id": "https://jopcmelo.dev/articles/binary-search", "content_html": "Imagine you're rummaging through a phonebook, looking for a name that begins with "L". Most likely, you wouldn't start from the first page and flip through each one until you reach "L", right? A more plausible approach would be to open the phonebook around the middle and move to the left or right depending on where "L" lies.
\nSimilarly, when searching for a word that starts with "P" in a dictionary, you'd probably start from the middle and navigate left or right, depending on the first letter of the word you're seeking.
\nWelcome to the world of binary search.
\n![\"Binary](\"/_next/image?url=%2F_next%2Fstatic%2Fmedia%2Fbinary-search-algorithm.1a798340.png&w=3840&q=75\"/)
\nAt the core, binary search is an algorithm designed to sniff out an element's location from a sorted list of elements. If the target element exists, the algorithm returns its location; if it doesn't, it returns None (or -1 in some programming languages).
For instance, consider you're seeking a number between 1 and 100. To guess, you must take the fewest possible attempts. Throughout the process, I'll let you know if your guess is too high, too low, or right on the money.
\nA straightforward approach (known as linear search) would be to take a shot in the dark, starting from 1, then 2, then 3, so on and so forth. If my number happens to be 99, you'd need 99 attempts to guess correctly. Would you consider that efficient? Probably not.
\nHowever, if you kick off by guessing 50 (the middlemost value), I'd nudge you in the right direction – up or down. If my number is higher than 50, you can write off all numbers between 1 and 50 and look only between 51 and 100. And if it's lower, you ditch numbers between 51 and 100, focusing only on 1 to 49. By continually halving possibilities, you home in on my number using the binary search algorithm.
\nHere's a glimpse of how many numbers binary search eliminates in each round: 50 -> 25 -> 13 -> 7 -> 4 -> 2 -> 1. In just 7 steps, you'd hit the jackpot.
\nNow, apply this logic to a dictionary with 480,000 words. Binary search lets you find a word in, at most, 19 steps!
\nBroadly speaking, a binary search needs log₂ n steps to return the correct value, whereas linear search requires n steps, where n corresponds to the list's length.
In case you feel a slight frown due to the term 'logarithms', here's a refresher with an emphasis on how to compute exponentials. The expression log₁₀ 100 asks: "how many instances of 10 can we multiply to reach 100?". The answer is 2 because 10 * 10 = 100. Therefore, log₁₀ 100 equals 2.
Note, however, that binary search works only if your list is sorted. What would ensue if the dictionary were not arranged in alphabetical order?
\nLet's go back to the linear search or a simple search where you check numbers sequentially. If you have a list of 100 numbers, worst-case scenario, you'd need 100 tries. For a list with 4 billion numbers, you'd need 4 billion attempts. The maximum tries are equivalent to the list's length, known as linear time.
\nBinary search differs. If your list has 100 items, you'd need at most 7 tries. If it contains 4 billion items, you need, at max, 32 tries. It's a potent algorithm, isn't it?
\nReady to see binary search in action? Here is a JavaScript implementation example to get you started:
\nfunction binarySearch(nums, item) {\n let left = 0\n let right = nums.length - 1\n\n do {\n let mid = Math.floor((left + right) / 2)\n let guess = nums[mid]\n\n if (guess === item) {\n return mid\n }\n\n if (guess > item) {\n // too high\n right = mid - 1\n } else {\n // too low\n left = mid + 1\n }\n } while (left <= right)\n\n return -1\n}\n\nconst result = binarySearch([1, 2, 3, 4, 5], 4) // 3\nconsole.log({ result })\nBy running this code, you'll see that it successfully locates the number "4" at the third index in the array. That's the magic of binary search in action! Keep in mind, this is a simple JavaScript implementation. Depending on the layout or type of your data, you may need to adjust it to suit your scenario best.
\nIn conclusion, binary search is an efficient algorithm that can save a lot of time and computing power when locating items in large, sorted datasets. It truly underscores the benefits of understanding your data and structuring it logically to optimize search operations. Happy coding, and may binary search be with you in your algorithmic journeys!
", "url": "https://jopcmelo.dev/articles/binary-search", "title": "Demystifying Binary Search: An Algorithm for Quick Discovery", "summary": "Explore the fascinating world of Binary Search. This piece digs deep into the inner workings of this efficient algorithm, illustrating its effectiveness against a straightforward linear search. With examples and a useful JavaScript implementation, the article simplifies Binary Search, making it accessible for all developers. Ideal read for those eager to optimize their search operations in large, sorted datasets.", "date_modified": "2023-10-20T00:00:00.000Z", "author": { "name": "João Melo" } }, { "id": "https://jopcmelo.dev/articles/ai-chat-with-langchain-pinecone-memory", "content_html": "In recent times, the evolution of AI has been permeating various sectors of businesses and the day-to-day lives of individuals. The applicability of artificial intelligence can be immensely useful in many aspects of programming, such as generating reports, deriving insights, automating processes, processing data, and most notably, in creating chatbots trained with specific data to behave in unique ways and provide contextually relevant responses.
\nWith this development, new terminology has become popular for those working with AI, including embeddings, vectors, contexts, LangChain, and more. In this article, we'll discuss how to create a chatbot using LangChain, Pinecone, and Upstash, with Next.js. The goal is to have an AI system ready to answer any question about an external context, store the history of message exchanges between the user and AI in a database, and use an intelligent search system to find the most appropriate response.
\nIn this article, we'll build our chatbot using Next.js, utilizing LangChain to manage conversation chains, Pinecone to handle a vector database for executing semantic similarity searches, and Upstash to store the history of message exchanges between the user and the AI.
\nAs we're about to create a project that leverages a vector database for embeddings and injects context into the AI to enable it to answer questions, it's important that you have an account with Pinecone.
\nYou'll need to follow the steps laid out by Pinecone to create and populate a vector index, as we'll be consuming it in this project. For this, you can follow the steps outlined in the Quickstart.
\nIn a real-world application, you'll need to devise a strategy for performing upsert operations on the vectors in your Pinecone index in accordance with your application's context. In this article, however, we'll focus more on how to consume the vector index for performing semantic similarity searches and passing context to the AI.
\nTo kick things off, let's create a Next.js project using the npx create-next-app command. We'll name our project demo-ai-chat:
npx create-next-app@latest\n![\"Creating](\"/_next/image?url=%2F_next%2Fstatic%2Fmedia%2Fnextjs-setup.892138c0.png&w=1920&q=75\"/)
\nNow, let's install the dependencies that we'll be using in this project. We'll install the following:
\nlangchain: a library for managing conversation chains@pinecone-database/pinecone: a library for dealing with a vector database and executing semantic similarity searches@upstash/redis: a library for managing the Redis databaseai: an interoperable, streaming-enabled, edge-ready software development kit for AI apps built with React and Svelte by Vercelpnpm install langchain @pinecone-database/pinecone @upstash/redis ai\nTo start, let's define a simple interface for the chat using the helper functions from the ai library provided by Vercel. Let's modify the app/page.tsx file with the following content:
'use client'\n\nimport { useChat } from 'ai/react'\n\nexport default function Chat() {\n const { messages, input, handleInputChange, handleSubmit } = useChat()\n\n return (\n <div className="stretch mx-auto flex w-full max-w-md flex-col py-24">\n {messages.length > 0\n ? messages.map((m) => (\n <div key={m.id} className="whitespace-pre-wrap">\n {m.role === 'user' ? 'User: ' : 'AI: '}\n {m.content}\n </div>\n ))\n : null}\n\n <form onSubmit={handleSubmit}>\n <input\n className="fixed bottom-0 mb-8 w-full max-w-md rounded border border-gray-300 p-2 shadow-xl"\n value={input}\n placeholder="Say something..."\n onChange={handleInputChange}\n />\n </form>\n </div>\n )\n}\nTo start, let's create a function to instantiate our Pinecone database. For this, let's create a file named lib/pinecone.ts with the following content:
import { PineconeClient } from '@pinecone-database/pinecone'\n\nexport const pinecone = new PineconeClient()\n\nexport const initPinecone = async () => {\n if (!process.env.PINECONE_API_KEY || !process.env.PINECONE_ENVIRONMENT) {\n throw new Error(\n 'PINECONE_API_KEY and PINECONE_ENVIRONMENT environment variables must be set'\n )\n }\n\n await pinecone.init({\n apiKey: process.env.PINECONE_API_KEY,\n environment: process.env.PINECONE_ENVIRONMENT,\n })\n}\nWe'll use this function to connect our application to Pinecone and perform semantic searches with LangChain. For now, let's set it aside and create a function to instantiate our Upstash database.
\nNow that we've managed to connect our application to Pinecone, let's create a function to instantiate our Upstash database. For this, let's create a file named lib/upstash.ts with the following content:
import { Redis } from '@upstash/redis'\n\nlet cachedDb: Redis\n\nexport async function getRedisClient() {\n if (cachedDb) {\n return cachedDb\n }\n\n if (!process.env.UPSTASH_ENDPOINT || !process.env.UPSTASH_TOKEN) {\n throw new Error(\n 'Please define the UPSTASH_ENDPOINT and UPSTASH_TOKEN environment variables inside .env.local'\n )\n }\n\n const client = new Redis({\n url: process.env.UPSTASH_ENDPOINT,\n token: process.env.UPSTASH_TOKEN,\n })\n\n cachedDb = client\n\n return client\n}\nWith this function, we can now connect our application to Upstash and save the history of message exchanges between the user and AI. For now, let's set it aside and focus on implementing LangChain, connecting to Pinecone, and saving the history of message exchanges between the user and AI in Upstash, all through a single chain. Sounds powerful, right? Let's get down to the heavy lifting.
\nNow that we've managed to connect our application to both Pinecone and Upstash, let's create an API to interact with the AI. For this, let's create a file named app/api/chat/route.ts with the following content:
import { StreamingTextResponse, LangChainStream } from 'ai'\nimport { ChatOpenAI } from 'langchain/chat_models/openai'\nimport { PineconeStore } from 'langchain/vectorstores/pinecone'\nimport { OpenAIEmbeddings } from 'langchain/embeddings/openai'\nimport { ConversationalRetrievalQAChain } from 'langchain/chains'\nimport { BufferMemory } from 'langchain/memory'\n\nimport { initPinecone, pinecone } from '@/lib/pinecone'\n\nexport const runtime = 'edge'\n\nexport async function POST(req: Request) {\n const { messages } = await req.json()\n\n const { stream, handlers } = LangChainStream()\n\n await initPinecone()\n\n const pineconeIndex = pinecone.Index('my-index')\n const vectorStore = await PineconeStore.fromExistingIndex(\n new OpenAIEmbeddings(),\n {\n pineconeIndex,\n namespace: 'my-namespace',\n textKey: 'content',\n }\n )\n\n const mainLLM = new ChatOpenAI({\n streaming: true,\n modelName: 'gpt-4',\n callbacks: [handlers],\n })\n\n const underlyingLLM = new ChatOpenAI({\n modelName: 'gpt-4',\n })\n\n const chain = ConversationalRetrievalQAChain.fromLLM(\n mainLLM,\n vectorStore.asRetriever(),\n {\n memory: new BufferMemory({\n memoryKey: 'chat_history',\n inputKey: 'question',\n outputKey: 'text',\n returnMessages: true,\n }),\n questionGeneratorChainOptions: {\n llm: underlyingLLM,\n },\n }\n )\n\n chain\n .call({\n question: messages[messages.length - 1].content,\n })\n .catch(console.error)\n\n return new StreamingTextResponse(stream)\n}\nHere we need to pay attention to some important points:
\nIn the above code, you'll need to replace YOUR_NAMESPACE_HERE with the namespace of your Pinecone index. You can find your Pinecone index's namespace on the index page within Pinecone.
textKeyIn the above code, you'll need to replace the content value with the value of the field you used to save the text of your document in your Pinecone index. When performing the vector upsert in Pinecone, you can define some metadata. It's important to save the content of each vector in the metadata so you can perform a semantic search using the created vectors and retrieve the document content. This key will be used internally by LangChain, accessing the vector metadata to retrieve the document content.
Depending on the purpose of your AI chatbot, you may need the AI to perform more complex tasks, so you might choose to use a more powerful model like gpt-4 or a faster and simpler model like gpt-3.5-turbo. In this example, we'll proceed with gpt-4 and end up using both later on.
Vamos ver o que temos construído até então:
\n![\"Demo](\"/_next/image?url=%2F_next%2Fstatic%2Fmedia%2Ffirst-demo.52b58243.gif&w=1200&q=75\"/)
\nWith what we have built so far, we have a chat with an AI that has interesting capabilities, such as responding to questions about a context that is indexed in the Pinecone vector database. Let's review what we have so far step by step.
\ngpt-4.One thing that may confuse you is the third step of the chain we created. What exactly does it mean for the LangChain to refactor the user's question? Let's understand this better.
\nImagine we have the following sequence of messages exchanged between the user and the AI:
\nUser: Who is Guillermo Rauch?\nAI: Guillermo Rauch is an Argentine programmer and the CEO of Vercel.\nUser: Where does he live?\nAI: He lives in San Francisco.\nNotice how the AI answered the user's question based on the context it has about Guillermo Rauch. Now, imagine having only the following question in the history:
\nUser: Where does he work?\nThe AI would not know that the pronoun he refers to Guillermo Rauch and therefore couldn't provide information about him.
That's why the LangChain refactors the user's question based on the memory history. This way, the AI can understand the context of the question and respond accordingly. With this intelligence, the AI can refactor the final user's question to something like:
\nUser: Where does Guillermo Rauch work?\nHere's a real example of how the AI refactors the user's question:
\n![\"AI](\"/_next/image?url=%2F_next%2Fstatic%2Fmedia%2Frephrased-question-by-ai.6334af5c.png&w=3840&q=75\"/)
\nMy original question was "How old is he?," but the AI refactored it to "How old is João?" so that the context search in Pinecone would be more effective.
\nNow that we understand how the LangChain uses the AI to refactor the question, it is easier to understand why we need to create two AI models. One is used to generate the final answer, while the other is used for the step of refactoring the user's question. Since we want only the final answer to be returned through streaming, we provide the streaming: true configuration only to the mainLLM, which is the model that will generate the final answer.
Remember that the step of refactoring the user's question occurs only when there is some history of messages exchanged between the user and the AI. If there is no history, the AI will not refactor the user's question. This means that for the first question asked to the AI, this step won't occur because there is no history. However, starting from the second question, this step will happen since there is already history.
\nWith the current version of our chat, we have something that can provide some value but will encounter some problems. Let's see what these problems are and how we can solve them.
\nSince we are using only BufferMemory in our chain, it means that we keep the message history only in memory. This implies that with every new request, the message history is lost. This becomes a problem because we cannot leverage the context of the message history to refactor the user's question.
One solution could be to use the messages variable we receive in the request to build the message history. However, that is not a good idea because the messages variable is just an array of messages stored in the user's browser. This means that the user can easily manipulate the message history or completely lose the message history by closing the chat page. Yes, we can solve this by using some form of disk storage on the user's machine, such as localStorage, but it is still not the ideal scenario because the message history of that user would be available only in that browser and computer, not across all devices the user uses.
\n\n"But João, what if I store the message history in a database?" - You.
\n
Yes, that's a good idea. Let's see how we can solve the first problem of our chain.
\nUpstashRedisChatMessageHistoryLangChain provides us with a memory implementation that stores the message history in external databases. I recommend you take a look at the options they offer. In this article, we will use UpstashRedisChatMessageHistory, which stores the message history in Upstash, a Redis database that is extremely performant and has low latency.
All we need to do is:
\nUpstashRedisChatMessageHistory from langchain.UpstashRedisChatMessageHistory.UpstashRedisChatMessageHistory instance to the BufferMemory in our chain.And that's it! Here's how our code looks:
\n// ...your other imports\nimport { UpstashRedisChatMessageHistory } from 'langchain/stores/message/upstash_redis'\n\nimport { getRedisClient } from '@/lib/upstash'\n\n// ...your code\n\nconst chatHistory = new UpstashRedisChatMessageHistory({\n sessionId: 'chat-session-1', // you can use the user id here, for example\n client: await getRedisClient(),\n})\n\nconst chain = ConversationalRetrievalQAChain.fromLLM(\n mainLLM,\n vectorStore.asRetriever(),\n {\n memory: new BufferMemory({\n memoryKey: 'chat_history',\n inputKey: 'question',\n outputKey: 'text',\n returnMessages: true,\n chatHistory,\n }),\n questionGeneratorChainOptions: {\n llm: underlyingLLM,\n },\n }\n)\n\n// ...your code\nNow, we can see that the message history is stored in Upstash, not in memory. This means that the message history is not lost with every new request, and we can leverage the context of the message history to refactor the user's question.
\n![\"Upstash](\"/_next/image?url=%2F_next%2Fstatic%2Fmedia%2Fupstash-screenshot.15efd332.png&w=3840&q=75\"/)
\nNow that we have solved the first problem, let's look at the second problem we face with our chain. Let's see what happens when we ask a question to the AI:
\n![\"GIF](\"/_next/image?url=%2F_next%2Fstatic%2Fmedia%2Fsecond-demo.e041a033.gif&w=1200&q=75\"/)
\nThe delay before starting to write a response is due to two main factors:
\nIn the semantic search, there isn't much we can do. But in the refactoring step of the user's question, we can make a small change to make the process faster.
\nLet's change the model we are using in underlyingLLM to gpt-3.5-turbo. This model is faster and simpler, and it will help us refactor the user's question more quickly.
// ...your code\n\nconst underlyingLLM = new ChatOpenAI({\n modelName: 'gpt-3.5-turbo',\n})\n\n// ...your code\nSince this step does not require such a powerful model because it involves simple reasoning, we can use a simpler and faster model, and thus speed up the process.
\nTake a look at what happens when we ask the AI a question about something mentioned in the message history:
\n![\"GIF](\"/_next/image?url=%2F_next%2Fstatic%2Fmedia%2Fdemo-with-wrong-history-context.68adc81e.gif&w=1200&q=75\"/)
\nThis happens because in the step of refactoring the user's question, LangChain provides instructions only to refactor the question and make it clearer and more concise. This means that if the person is mentioning any information that was said in the message history, that information is not effectively utilized to refactor the user's question.
\nTo address this, we can make a small adjustment to the AI instruction in this step to consider the relevant portion of the message history according to the user's question, if there is any relevant portion.
\nTo make this adjustment, we create a new constant with a prompt that has clearer and stronger instructions, which will prompt the AI to use relevant excerpts from the message history that may be useful in answering the question. This way, the AI will not solely depend on the context obtained from Pinecone but also consider any important excerpts from the message history. Here's how our code looks like:
\n// ...your code\n\nconst QA_UNDERLYING_PROMPT_TEMPLATE = `Given the following conversation and a follow up question, return the conversation history excerpt that includes any relevant context to the question if it exists and rephrase the follow up question to be a standalone question.\nChat History:\n{chat_history}\nFollow Up Input: {question}\nYour answer should follow the following format:\n\\`\\`\\`\nUse the following pieces of conversation context to answer the users question.\nIf you don't know the answer, just say that you don't know, don't try to make up an answer.\n----------------\nRelevant chat history excerpt:\n<Relevant chat history excerpt as context here or "N/A" if nothing found>\n\nQuestion: <Rephrased question here>\n\\`\\`\\`\nYour answer:\n`\n\n// ...your code\n\nconst chain = ConversationalRetrievalQAChain.fromLLM(\n mainLLM,\n vectorStore.asRetriever(),\n {\n memory: new BufferMemory({\n memoryKey: 'chat_history',\n inputKey: 'question',\n outputKey: 'text',\n returnMessages: true,\n chatHistory,\n }),\n questionGeneratorChainOptions: {\n llm: underlyingLLM,\n template: QA_UNDERLYING_PROMPT_TEMPLATE,\n },\n }\n)\n\n// ...your code\nWith this change, we are telling the AI that if there is any relevant excerpt in the message history, it should be used to answer the user's question. Here's the result:
\n![\"GIF](\"/_next/image?url=%2F_next%2Fstatic%2Fmedia%2Ffixed-history-context-issue-screenshot.fb637ea7.png&w=3840&q=75\"/)
\nAs your users continue to use your chat, the memory of each chat session becomes increasingly larger. This means that with each new request, the LangChain will include the message history in the context of the AI, which can cause you to exceed the AI's token limits. To address this scenario, we have a few options:
\nBufferMemory, you can use other memory implementations provided by LangChain to, for example, keep only the most recent N messages or always summarize the message history.If you want to offer a good user experience, I recommend exploring the memory options that LangChain provides. The tool offers ways to optimize this.
\nNow that we have addressed all the problems encountered with our chain, we have a well-functioning chat system that can provide significant value to our users. Our AI has the main optimizations we need to deliver a good user experience, with valid and contextualized responses, access to previous conversation memory, and good performance.
\nHere's the final code of our chat:
\nimport { StreamingTextResponse, LangChainStream } from 'ai'\nimport { ChatOpenAI } from 'langchain/chat_models/openai'\nimport { PineconeStore } from 'langchain/vectorstores/pinecone'\nimport { OpenAIEmbeddings } from 'langchain/embeddings/openai'\nimport { UpstashRedisChatMessageHistory } from 'langchain/stores/message/upstash_redis'\nimport { ConversationalRetrievalQAChain } from 'langchain/chains'\n\nimport { initPinecone, pinecone } from '@/lib/pinecone'\nimport { getRedisClient } from '@/lib/upstash'\nimport { BufferMemory } from 'langchain/memory'\n\nexport const runtime = 'edge'\n\nconst QA_UNDERLYING_PROMPT_TEMPLATE = `Given the following conversation and a follow up question, return the conversation history excerpt that includes any relevant context to the question if it exists and rephrase the follow up question to be a standalone question.\nChat History:\n{chat_history}\nFollow Up Input: {question}\nYour answer should follow the following format:\n\\`\\`\\`\nUse the following pieces of conversation context to answer the users question.\nIf you don't know the answer, just say that you don't know, don't try to make up an answer.\n----------------\nRelevant chat history excerpt:\n<Relevant chat history excerpt as context here or "N/A" if nothing found>\n\nQuestion: <Rephrased question here>\n\\`\\`\\`\nYour answer:\n`\n\nexport async function POST(req: Request) {\n const { messages } = await req.json()\n\n const { stream, handlers } = LangChainStream()\n\n await initPinecone()\n\n const pineconeIndex = pinecone.Index('my-index')\n const vectorStore = await PineconeStore.fromExistingIndex(\n new OpenAIEmbeddings(),\n {\n pineconeIndex,\n namespace: 'demo-ai-chat',\n textKey: 'content',\n }\n )\n\n const slowerModel = new ChatOpenAI({\n streaming: true,\n modelName: 'gpt-4',\n callbacks: [handlers],\n })\n\n const fasterModel = new ChatOpenAI({\n streaming: false,\n modelName: 'gpt-3.5-turbo',\n })\n\n const chatHistory = new UpstashRedisChatMessageHistory({\n sessionId: 'chat-session-1',\n client: await getRedisClient(),\n })\n\n const chain = ConversationalRetrievalQAChain.fromLLM(\n slowerModel,\n vectorStore.asRetriever(),\n {\n memory: new BufferMemory({\n memoryKey: 'chat_history',\n inputKey: 'question',\n outputKey: 'text',\n returnMessages: true,\n chatHistory,\n }),\n questionGeneratorChainOptions: {\n llm: fasterModel,\n template: QA_UNDERLYING_PROMPT_TEMPLATE,\n },\n }\n )\n\n chain\n .call({\n question: messages[messages.length - 1].content,\n })\n .catch(console.error)\n\n return new StreamingTextResponse(stream)\n}\nIf you want to see the code in action, you can access the GitHub repository of this project.
\nIn this article, we've learned how to create a chatbot using LangChain, Pinecone, and Upstash, with Next.js and streaming. We've learned how to create a chain with LangChain, connect to Pinecone, and save the history of message exchanges between the user and AI in Upstash, all through a single chain. We've also learned how to use the UpstashRedisChatMessageHistory to save the history of message exchanges between the user and AI in Upstash, and how to use the gpt-3.5-turbo model to speed up the process of refactoring the user's question. Also, we've walked through how to optimized the chain to take advantage of the relevant context of the history of message exchanges between the user and AI.
I still have a great optimization I'd like to share with you, but I'll leave that for the next post. This optimization makes context injection into AI brought from Pinecone more effective and we just use the amount of context needed without consuming more tokens than necessary to answer the user's query. It will be a very interesting article, so stay tuned for it!
\nI hope you enjoyed this article and that it helps you create your own AI chatbot. If you have any questions, feel free to reach out to me on Twitter at @joaomelo.
", "url": "https://jopcmelo.dev/articles/ai-chat-with-langchain-pinecone-memory", "title": "How to create an AI chat chain with LangChain, Pinecone, and memory", "summary": "Master the art of AI chat chains with LangChain, Pinecone, and Upstash, using Next.js. Elevate communication, harness the potential of intelligent conversations, and unlock the future of interaction.", "date_modified": "2023-10-16T00:00:00.000Z", "author": { "name": "João Melo" } }, { "id": "https://jopcmelo.dev/articles/understanding-aws-lambda-a-comprehensive-guide-to-serverless-computing-for-everyone", "content_html": "Are you looking for an easy-to-understand, comprehensive guide to serverless computing? You've come to the right place! In this article, we will explore AWS Lambda, one of the most popular serverless computing services. With AWS Lambda, you can build applications without worrying about managing servers or infrastructure. By the end of this post, you'll have a solid understanding of AWS Lambda and how it can benefit your projects. Let's dive in!
\nServerless computing is a cloud computing model that allows you to build and run applications without managing servers. Instead, you focus on writing code, while the cloud provider (in this case, AWS) takes care of the underlying infrastructure, scaling, and maintenance. This enables you to develop faster, reduce costs, and improve resource utilization.
\nIn recent years, serverless computing has gained significant popularity due to its simplicity, flexibility, and cost-effectiveness. By eliminating the need to manage servers, developers can concentrate on their core competencies and deliver applications quickly and efficiently.
\nAWS Lambda is a serverless compute service offered by Amazon Web Services (AWS). It allows you to run your code in response to events such as HTTP requests, changes in a database, or file uploads. Lambda automatically manages the compute resources, so you only pay for the actual execution time of your functions.
\nAWS Lambda provides developers with several benefits:
\nTo get started with AWS Lambda, you'll need to follow these general steps:
\nEach step involves specific actions and configurations that you'll need to understand and implement. There are plenty of resources and tutorials available online to help you through the process.
\nAWS Lambda can be used for a variety of tasks and applications. Here are some common use cases:
\nTo make the most out of AWS Lambda, it's essential to follow some best practices:
\nAWS Lambda is a powerful and flexible solution for building serverless applications. By leveraging its automatic scaling, cost-effectiveness, and event-driven capabilities, you can focus on writing code while AWS handles the operational aspects. With a better understanding of AWS Lambda and its various use cases, you're now ready to embark on your serverless journey! Don't forget to share this article with your friends and colleagues who might find it useful as well. Happy coding!
", "url": "https://jopcmelo.dev/articles/understanding-aws-lambda-a-comprehensive-guide-to-serverless-computing-for-everyone", "title": "Understanding AWS Lambda: A comprehensive guide to serverless computing for everyone", "summary": "A comprehensive guide to serverless computing with AWS Lambda, including key features, use cases, and best practices for building serverless applications.", "date_modified": "2023-03-15T00:00:00.000Z", "author": { "name": "João Melo" } }, { "id": "https://jopcmelo.dev/articles/how-to-create-a-winning-professional-profile-on-linkedin-github-and-stack-overflow", "content_html": "Creating a profile that stands out is crucial when searching for job opportunities, freelancing work, or building a professional network. Your profile is the first impression that potential employers or clients have of you, so it is important to ensure it is well-presented, relevant, and highlights your expertise. In this article, we will provide you with tips for creating a professional profile on LinkedIn, GitHub, and Stack Overflow.
\nLinkedIn is a social media platform designed for professionals to connect, showcase their work, and find job opportunities. Here are some tips to make your profile stand out:
\nGitHub is a platform for developers to host and review code, manage projects, and build software. Here are some tips to make your profile stand out:
\nIn today's highly competitive job market, having a strong and well-structured profile on professional networking sites like LinkedIn, GitHub, and Stack Overflow is more important than ever. Recruiters and hiring managers often turn to these platforms to find and evaluate potential job candidates. A concise and complete profile that showcases your skills, expertise, and accomplishments can help you stand out from the crowd and catch the attention of recruiters.
\nBy following the tips mentioned in this article, you can create a profile that highlights your strengths and achievements, presents your work in a professional and organized manner, and positions you as a valuable asset to potential employers. Whether you are a software developer, data scientist, or business professional, having a strong presence on these platforms can help you connect with other professionals, build your network, and ultimately land your dream job.
\nSo, take the time to optimize your profile on LinkedIn, GitHub, and Stack Overflow by adding a profile picture, bio, and your work history. Show off your skills and experience by sharing your projects and contributing to open source projects. Keep your profile up-to-date with the latest information and engage with the community to build your reputation and expand your network. With a little effort and attention, you can create a profile that gets you noticed by recruiters and helps you achieve your career goals.
", "url": "https://jopcmelo.dev/articles/how-to-create-a-winning-professional-profile-on-linkedin-github-and-stack-overflow", "title": "How to create a winning professional profile on LinkedIn, GitHub, and Stack Overflow", "summary": "Tips for creating a professional profile on LinkedIn, GitHub, and Stack Overflow", "date_modified": "2023-02-24T00:00:00.000Z", "author": { "name": "João Melo" } }, { "id": "https://jopcmelo.dev/articles/is-nodejs-single-threaded-or-not", "content_html": "There is a very common question in the NodeJS developer community about whether NodeJS is single-threaded. We first need to understand how NodeJS works underneath the scenes to find the answer.
\nLet's get started!
\nThe Event Loop is a built-in feature of NodeJS built around a cycle of events. This is how NodeJS executes all instructions in its algorithm and passes these instructions to the machine processor.
\nEvent Loop is single-threaded. That is, all instructions are sent to a single thread, called the Event Queue, and then either each instruction in that list is processed in order, or they are handed off to another thread (Thread Pool) that will execute them in the background. This second thread that executes instructions in the background is made possible by libuv (a C library originally written for NodeJS to abstract non-blocking input/output operations).
\nAs soon as each Event Loop operation is finished, NodeJS returns to the list of next tasks and focuses on executing the next non-blocking task.
\nNodeJS is event-based (like requests, calls, etc). All these events are processed primarily in a single thread and may vary according to the type of operation (such as whether it is blocking I/O or not).
\nThe step-by-step that NodeJS performs are:
\n![\"Event](\"/_next/image?url=%2F_next%2Fstatic%2Fmedia%2Fimage-1.484c92da.webp&w=3840&q=75\"/)
\nThe Thread Pool (where slower operations are executed in the background) has 4 threads by default, but this can be configured by hand to have varying threads. In other words, the Thread Pool is not single-threaded but is part of the NodeJS core.
\nThe Thread Pool does not create a new thread for each operation but rather shares a pool of threads. This makes NodeJS more efficient concerning the consumption of computer resources. Because of this, NodeJS will use much less computational resources than an application made in PHP or .NET, for example (This does not mean that NodeJS is better or worse than PHP or .NET).
\n1 request on Google's website executes about 88 million instructions on the CPU. So NodeJS delegates to someone else (Thread Pool) to execute this instruction so that while this is happening in the background, NodeJS can execute another 88 million instructions in parallel through the Event Loop instead of blocking 100% of the operations.
\nOr when reading 1KB of data from a file, it takes 1.4 microseconds on an SSD that reads 200-730MB/s. On a 2GHz processor, this results in an execution of 28,000 instructions. Because of this, NodeJS does not block consumption and keeps this heavier operation in a background process.
\nNodeJS is single-threaded, but it relies on the help of the libuv library to perform some multi-threaded operations in the background.
", "url": "https://jopcmelo.dev/articles/is-nodejs-single-threaded-or-not", "title": "Is NodeJS single threaded or not?", "summary": "After all, what is the truth about this discussion? Let's discuss how NodeJS works behind the scenes and draw a conclusion about it", "date_modified": "2022-02-15T00:00:00.000Z", "author": { "name": "João Melo" } }, { "id": "https://jopcmelo.dev/articles/how-to-work-remotely-while-earning-in-dollars", "content_html": "The subject of this publication may seem an impossible or very difficult reality to achieve, right? It did for me too. Today I will share how I became an international developer and give some golden tips that would have helped me absurdly if I had received them while working in my home country.
\nRemote work became a strong reality for us programmers a while ago because of this difficult time we are going through in history, however along with it came some problems, such as:
\nWith these challenges coming all at once, we need to be careful and keep a close eye on things, so we don't develop burnout from overwork, for example. But what is the relationship between the problems that the pandemic has brought to us and remote work? A company has to know how to deal with remote work to energize its community and achieve incredible milestones without putting its employees through delicate and bad situations. I started following X-Team a little over 1 year ago in this relationship.
\nX-Team is a 100% remote company that has been improving its community for 15 years. X-Team hires people worldwide to form high-quality engineering teams and allocates these teams to large companies like FOX, Riot Games, Coinbase, Twitter, Dell, etc. X-Team could stop at the part about assigning teams to their customers, but they go much further than that, following their philosophy:
\n\n\n"Live your life like a great adventure again because when you can work from anywhere and you commit to using that flexibility to do more of what energizes you." - Ryan Chartrand, CEO of X-Team.
\n
The company invests heavily in the employee community to ensure everyone works happily, is energized, and is motivated. It is a fact that when someone works happy, energized, and motivated; consequently that person will deliver the best version of their work. But how do I feel energized? Let's talk about some of the X-Team's key incentive methods.
\nUnleash+ is a $2,500 credit that X-Team gives you once you join the team to spend on a variety of categories, whether it's buying equipment for a podcast, buying an accessory for your bike, paying for therapy sessions, buying a course, or even building your own smart home. You use this credit to spend on things that will energize you and make you evolve more and more as a person.
\nThis credit renews every year, so as long as you are an X-Teamer, you can use this credit to energize yourself and evolve.
\nThe Vault is the X-Team's store for exclusive and collectible items. You can exchange coins for exclusive items such as t-shirts, sweatshirts, mugs, backpacks, and hats. The most amazing thing about it is that these items are so whimsical that you always want to buy as many as possible.
\nBut how to get coins? There are some "challenges" called bounties on the X-Team's collaborator platform. They are very diverse, ranging from "Eat a pizza on a Saturday" to "Cycle 100km". When you complete a challenge, take a picture or screenshot and share it on the company's Slack server, and you will be rewarded with some coins directly in your balance to spend in The Vault.
\nBest of all, the items are renewed following a theme of seasons that last around 3 months. In other words, every 3 months, there are new exclusive, and amazing collectibles.
\nX-Outposts are the most famous events of the X-Team community. They organize monthly hacker houses worldwide so X-Teamers can meet, share experiences, and see new places worldwide. These events have occurred in some amazing places like Costa Rica, Austria, Bali, Mexico, Japan, and South Africa.
\nX-Team has several clients that can place you in their internal team, like Riot Games, FOX, Coinbase, Twitter, Dell, etc.
\nBesides large companies as clients, the opportunities to work with technologies on the rise in the market, such as React Native, ReactJS, VueJS, NodeJS, Python, Go, Scala, Serverless, and various other technologies.
\nX-Team is a rapidly growing company, which also makes the demand for new developers within the community great. In other words, the company is constantly hiring programmers for various positions.
\nThe company has a platform where you can create an account, fill in some data about your career, and apply for any available position. However, as said before, X-Team is growing a lot and consequently receives thousands of applications weekly, making it difficult for the recruiting team to see them. Because of this, in this part of this post, I will explain how the X-Team recruitment process works and also give you some tips on how to become a top candidate for the X-Team recruitment team.
\nOnce you have created an account on the X-Team vacancy platform, filled out your profile, and applied for a vacancy, the process starts with an introductory meeting where the X-Team recruitment team will talk to you by video call to understand more about your career moment, your plans and your projects. They will also introduce how the "X-Team world" works and answer any questions you might have.
\nHere are some tips:
\nIt is also very important to know how to present yourself to avoid mistakes that can make you fail to move on to the next stage, such as bad lighting, bad audio, and improper dress.
\nAfter passing the first screening and X-Team identifying that you have a cultural fit with the company, a second stage will be scheduled, defined by a technical interview, but rest assured; there is usually no live coding, but usually, some technical questions focused on the technologies of the position you applied for.
\nGoing through the technical interview, X-Team will present your profile to their clients, and you may have some technical meetings with some clients, as they usually want to get to know you and ask more targeted questions for their work stack.
\nSuppose any client wants you on their team; congratulations! X-Team will contact you to finalize some contractual processes, and you will have a start date lined up with the recruitment team.
\nIf no client is sure they want you on their team, don't worry; if X-Team feels confident in you and feels there is a good cultural match between you and X-Team, you may receive an offer to work on the company's internal team. In contrast, searching for a client for you continues in the background, and you will follow the same contractual process.
\nIf none of the above scenarios occur, there is nothing to worry about! You can work on some issues to increase your chances of receiving an X-Team invitation and improve the information you provided when creating your account on the X-Team job platform. Always update your profile; it will help you find an opportunity that is right for you!
\nRemember that you can contact X-Team anytime with your questions or even ask them during a video call. However, X-Team has already prepared the main questions about the company; this can help you understand more about the company before you even start an interview.
\nI hope this post has answered most of the questions of the people who have contacted me asking for help. The fact that there is now a post about how I joined X-Team and giving some tips for getting ahead doesn't mean that you can't contact me via social media (at the bottom of this platform) for any questions. I'll be happy to talk to you 🙂 I'll be glad to talk to you.
\nIn case this post has helped you in any way, I strongly urge you to share it with a friend who desires to work remotely, earning in dollars but doesn't know how or has some doubts; it would be of great help.
", "url": "https://jopcmelo.dev/articles/how-to-work-remotely-while-earning-in-dollars", "title": "How to work remotely while earning in dollars", "summary": "How I became an international developer, and some golden tips that would have helped me absurdly if I had received them while working in my home country.", "date_modified": "2021-08-10T00:00:00.000Z", "author": { "name": "João Melo" } }, { "id": "https://jopcmelo.dev/articles/optimizing-react-components", "content_html": "To understand when we should use the memo, we first need to understand how React works with rendering.
\nmemo for?Since you already know how React works under the hood, let's move on to the applicability of the memo and take our applications to the next level. The memo will prevent React from re-rendering the child component if it hasn't changed when the parent component changes.
A clear example of the problem that memo solves in React is: whenever a parent component undergoes a change (whether in properties, state, css, etc.), the child components are also rendered again.
Now that we know what the memo is for within the React world, let's go for a hands on!
Let's take the following code as an example:
\n// pages/Home.tsx\n\nimport { useState } from 'react';\nimport { api } from '../services/api';\nimport { Item } from '../components/Item';\n\nfunction List() {\n\tconst [items, setItems] = useState([]);\n\n\tasync function handleClick() {\n\t\tconst response = await api.get('/items')\n\t\tsetItems(response);\n\t};\n\n\treturn (\n\t\t<div>\n\t\t\t<div>\n\t\t\t\t{items.map((item) => (\n\t\t\t\t\t<Item key={item.title} item={item} />\n\t\t\t\t))}\n\t\t\t</div>\n\n\t\t\t<button type="button" onClick={handleClick}>Load items</button>\n\t\t</div>\n\t);\n};\n// components/Item.tsx\n\nimport { memo } from 'react';\n\ninterface ItemComponentProps {\n\titem: {\n\t\ttitle: string;\n\t}\n};\n\nfunction ItemComponent({ item }: ItemComponentProps) {\n\treturn (\n\t\t<div>\n\t\t\t<h1>{item.title}</h1>\n\t\t</div>\n\t)\n};\n\nexport const Item = memo(ItemComponent, (prevProps, nextProps) => {\n\treturn Object.is(preProps.item, nextProps.item);\n});\nIn this example, the following scenario takes place:
\n<Item /> for each record returned from the API;memo, React will behave as fallows when it starts the process of creating the Virtual DOM for each child component:\nDid you know that the code snippet {} === {} returns false in JavaScript? This is because the equality operator === considers the comparison of memory and not just of value. That is, if you compare two objects that occupy different memory locations but have the same value, JavaScript will identify that they are different because they occupy different memory locations. For that reason, the memo can still re-render the child components since prevProps, and next props can occupy different memory locations.
To solve this problem, memo gets a second parameter, a function in which we will define what it means for a property to have been changed or not for React. With this, we use Object.is() to do a deep comparison considering only the values of the object we received as a property in the component. If all values are equal, our function will return a true value, and React will not render the child component unnecessarily.
Don't use memo prematurely without real need. Unnecessary use of memo can cause your application to lose performance because it has to execute all the memo logic even when it is unnecessary.
Memo is used in some main situations:
useMemo hookThe use of useMemo prevents unnecessary heavy processing from being re-run along with the re-rendering of the component. Let's see an example of how we can apply this hook!
Let's assume the following code:
\ninterface CartShoppingProps {\n\titems: {\n\t\tid: string;\n\t\tprice: string;\n\t};\n};\n\nexport function CartShopping({ items }: CartShoppingProps) {\n\tconst total = items.reduce((accumulator, item) => {\n\t\treturn accumulator + item.price;\n\t}, 0);\n\n\treturn (\n\t\t<h1>Total price: {total}</h1>\n\t);\n};\nIf some change causes the <CartShopping /> component to be re-rendered, reduce() will be executed again even if the final value is the same;
With useMemo, React avoids running reduce() again if the result remains the same. See the example below:
import { useMemo } from 'react';\n\ninterface CartShoppingProps {\n items: {\n id: string;\n price: string;\n };\n};\n\nexport function CartShopping({ items }: CartShoppingProps) {\n const total = useMemo(() => {\n return items.reduce((accumulator, item) => {\n return accumulator + item.price;\n }, 0);\n })\n\n return (\n <h1>Total price: {total}</h1>\n );\n};\nIn this scenario, if the component is re-rendered, but the values remain in the properties, reduce() will not run again, and the application will consume less processing;
With useMemo, React avoids a variable occupying a new memory location when we use it to pass on to a child component;
useCallback hookWe use useCallback when we want to remember a function, not a value. For values we use useMemo.
When we create a function and this function is going to be passed on to the child components of our application, it is important that this function uses useCallback because every time a function is re-created in memory, React will re-render the component that receives the function, even if it has not changed.
The useCallback receives 2 parameters:
useCallback hook;Even though we use useMemo to avoid unnecessary calculations, React has comparison logic to identify whether calculations should be redone or not through useMemo. However, if I know when I should do the calculations, it is more interesting for me not to use useMemo so that it is possible to format the data at the interface build level and perform this only when I know we will need it (when getting the response from an API, for example).
When you generate the build of your React application, the process is: to take the entire import file tree and implement it in your bundle. In other words, the bundle contains all the code required for our application to run.
\nHowever, there is a scenario where we only want to import a file if some condition is met (e.g., display a formatted date if the user clicks to open a modal). This becomes a problem because the bundle code will include code that will be used only in a specific condition, and not always.
\nTo solve this problem we can do lazy loading only when we need it. Here is some sample code for this implementation:
\nFor applications with pure ReactJS:
\nimport { lazy as lazyLoading } from 'react';\nFor applications with NextJS:
\nimport lazyLoading from 'next/dynamic';\nconst MyLazyComponent = lazyLoading(() => {\n\treturn import('../components/MyComponent');\n }, {\n loading: () => <span>Loading...</span>, // here you can define a loading component to be shown when the application is "downloading" the component to be shown.\n });\nAnd now, just use your component, and it will be loaded only when it actually needs to be used.
\nLet's imagine a list of 1,000 items. Once a user accesses this list, it will take a lot of effort for the browser to render all the items in the list. Considering that not all 1,000 items will fit inside the end user's browser view, we can work with a virtualization solution, in which React will display only the elements that the user has a field-of-view view of, thus drastically reducing the processing to display the full list.
\nFor this, I like to use a library called react-virtualized. So let's go through step by step how to implement this functionality!
\nInstall the library:
\nyarn add react-virtualized && yarn add @types/react-virtualized -D\nCreate a component that will be responsible for rendering the list:
\nimport { List, AutoSizer, ListRowRenderer } from 'react-virtualized';\n\ninterface MyComponentProps {\n\titems: string[];\n};\n\nexport function MyComponent({ items }: MyComponentProps) {\n\tconst rowRenderer: ListRowRenderer = ({ index, key, style }) => {\n\t\treturn (\n\t\t\t<div key={key} style={style}>\n\t\t\t\t<li>{items[index]}</li>\n\t\t\t</div>\n\t\t);\n\t};\n\n\treturn (\n\t\t<AutoSizer>\n\t {({height, width}) => (\n\t <List\n\t height={height}\n\t width={width}\n\t rowCount={list.length}\n\t rowHeight={20}\n\t rowRenderer={rowRenderer}\n\t\t\t\t\toverscanRowCount={5} // prefetch next N items to be shown\n\t />\n\t )}\n\t </AutoSizer>\n\t);\n};\nWith this, we will always show the number of items needed to fill the client's screen, and we will always have the next 5 items pre-loaded to be displayed if the user scrolls to see more records. This makes the rendering of many elements more performant.
\nWhen we generate our build for production, it can get very heavy and get in the way of other steps in the deployment. But, how can we solve this? How about we analyze the size of our bundle and identify which modules have the largest size so we can act upon this and optimize our deployment? This is possible and it's very easy to apply in a NextJS application! Let's see how to do it in practice 😉
\nInstalling the library:
\nyarn add @next/bundle-analyzer\nIn the root of your project, create a file called next.config.js with the following information:
\nconst withBundleAnalyzer = require('@next/bundle-analyzer')({\n\tenabled: process.env.ANALYZE === 'true',\n});\n\nmodule.exports = withBundleAnalyzer({});\nNow, to analyze the bundle generated in your project build, just run the following command in your terminal:
\nANALYZE=true yarn build\nThe command will open a window in your browser to show you which packages weigh the most in your bundle.
\nI hope you have enjoyed this content and that you have learned something to take you to the next level and perform your projects.
\nThis post took a few hours to complete and if you have any questions, leave them in the comments below and I or any other member that knows the answer will answer them as soon as possible.
\nFinally, here are some references for you to further explore this performance-oriented content in React:
\n", "url": "https://jopcmelo.dev/articles/optimizing-react-components", "title": "Optimizing ReactJS components", "summary": "Understand how to optimize ReactJS components using hooks the right way.", "date_modified": "2021-06-19T00:00:00.000Z", "author": { "name": "João Melo" } }, { "id": "https://jopcmelo.dev/articles/understanding-react-renderings", "content_html": "Today we’re going to get a better understanding of the main rendering concepts within React. For that, we need to understand what are the main causes that cause a component to be rendered.
\nLet's start this journey!
\nWhenever the parent component has a new rendering (be it for any reason like change of state, context, called the API, etc ...), the child component will automatically be rendered.
\nWhen any component property has a changed value, that component is rendered again.
\nWhen we call a hook in which it has a data-altering effect (useState, useContext, useReducer, etc...), the component responsible for the call is also rendered again.
\nNow that we know the main reasons why React renders a component, let's understand how React does it.
\nLet's imagine that one of the situations mentioned above happened in a component. Now let's see the step by step that React would do to render the updated component:
\nAssuming we have the following list structure in a list component:
\n<ul>\n\t<li className="1">1</li>\n\t<li className="2">2</li>\n\t<li className="3">3</li>\n <!-- ...N -->\n\t<li className="N">N</li>\n</ul>\nImagine that a customer deleted the item from className="532". For React, when it does the new rendering, it will:
\nIf you don't already know the React DevTools browser extension, you need to know it now! This tool gives you "super powers" to analyze your ReactJS applications.
\nWe will address the main features that it provides us and why we should use them.
\nFirst of all, know that this extension exists for several browsers, such as Google Chrome, Firefox, Microsoft Edge, etc. So, I will leave the installation link only for Google Chrome (since it is the most used browser), but be aware that searching the internet you can find the installation link for your browser without any problems.
\nLink to install on Google Chrome
\nOnce installed on your browser, just your browser's developer console and you will notice that the extension added two new tabs: "Components" and "Profiler".
\nNow, let's cover each of the tabs added by the extension.
\nHere it is possible to inspect each element that we have in our applications.
\nWithin each component, we can see:
\nTo highlight components that are rendered in the Virtual DOM due to diffing checks, click on the gear icon → General → Enable the "Highlight updates when components render" option.
\n![\"Components](\"/_next/image?url=%2F_next%2Fstatic%2Fmedia%2Fimage-1.e648348e.webp&w=1200&q=75\"/)
\nIt serves to understand why a component was rendered. To enable this functionality, click on the gear icon → Profiler → Enable the "Record why each component rendered while profiling" option.
\n![\"Profiler](\"/_next/image?url=%2F_next%2Fstatic%2Fmedia%2Fimage-2.63d91e5e.webp&w=1080&q=75\"/)
\nBy default, the Profiler is recording the behavior of the components as soon as the page is loaded. To stop recording and see the results, we need to click on the recording icon (ball) at the top left of DevTools.
\nAfter stopping recording, we will see a result with the reasons why a component was rendered, see an example:
\n![\"Recording](\"/_next/image?url=%2F_next%2Fstatic%2Fmedia%2Fimage-3.a44052d5.webp&w=1080&q=75\"/)
\nThat's all, folks! Today we learn advanced rendering concepts in React and we also know React DevTools, a super powerful tool that we can use to measure performance and understand what happens behind the scenes in our React applications.
\nI hope you guys enjoyed this content and we’re leaving next week! Peace out!
", "url": "https://jopcmelo.dev/articles/understanding-react-renderings", "title": "Understanding React renderings", "summary": "Understand advanced React rendering concepts and learn powerful tools to analyze the performance of your application.", "date_modified": "2021-05-21T00:00:00.000Z", "author": { "name": "João Melo" } } ] }