![\"Essential](\"https:\/\/www.kdnuggets.com\/wp-content\/uploads\/wijaya-essential-python-libraries-1.png\")
\nImage generated with Midjourney<\/font><\/center>
\n <\/p>\n
As a data professional, it\u2019s essential to understand how to process your data. In the modern era, it means using programming language to quickly manipulate our data set to achieve our expected results.<\/p>
![\"NWU](\"https:\/\/www.kdnuggets.com\/wp-content\/uploads\/s-nwu-2605.jpg\")
\nAnalytics, AI, and leadership skills.\t\t<\/p>\n<\/a><\/div><\/div>\n
Python is the most popular programming language data professionals use, and many libraries are helpful for data manipulation. From a simple vector to parallelization, each use case has a library that could help.<\/p>
\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t![\"NVIDIA](\"https:\/\/www.kdnuggets.com\/wp-content\/uploads\/s-pny-2603.jpg\")
\nLearn more<\/p>\n<\/a><\/div><\/div>\n
So, what are these Python libraries that are essential for Data Manipulation? Let\u2019s get into it.<\/p>\n
<\/p>\n
1.NumPy<\/h2>\n
<\/p>\n
The first library we would discuss is NumPy<\/a>. NumPy is an open-source library for scientific computing activity. It was developed in 2005 and has been used in many data science cases.<\/p>\n NumPy is a popular library, providing many valuable features in scientific computing activities such as array objects, vector operations, and mathematical functions. Also, many data science use cases rely on a complex table and matrices calculation, so NumPy allows users to simplify the calculation process.<\/p>\n Let\u2019s try NumPy with Python. Many data science platforms, such as Anaconda, have Numpy installed by default. But you can always install them via Pip.<\/p>\n <\/p>\n After the installation, we would create a simple array and perform array operations.<\/p>\n <\/p>\n Output: We can also perform basic statistics calculations with NumPy.<\/p>\n <\/p>\n The data mean:4.0, median:4.0, and standard deviation: 2.0<\/p>\n It\u2019s also possible to perform linear algebra operations such as matrix calculation.<\/p>\n <\/p>\n Output:<\/p>\n There are so many benefits you can do using NumPy. From handling data to complex calculations, it\u2019s no wonder many libraries have NumPy as their base.<\/p>\n <\/p>\n <\/p>\n Pandas<\/a> is the most popular data manipulation Python library for data professionals. I am sure that many of the data science learning classes would use Pandas as their basis for any subsequent process.<\/p>\n Pandas are famous because they have intuitive APIs yet are versatile, so many data manipulation problems can easily solved using the Pandas library. Pandas allows the user to perform data operations and analyze data from various input formats such as CSV, Excel, SQL databases, or JSON.<\/p>\n Pandas are built on top of NumPy, so NumPy object properties still apply to any Pandas object.<\/p>\n Let\u2019s try on the library. Like NumPy, it\u2019s usually available by default if you are using a Data Science platform such as Anaconda. However, you can follow the Pandas Installation guide<\/a> if you are unsure.<\/p>\n You can try to initiate the dataset from the NumPy object and get a DataFrame object (Table-like) that shows the top five rows of data with the following code.<\/p>\n Then you can try several data manipulation activities, such as data selection.<\/p>\n <\/p>\n It\u2019s possible to do the Data calculation.<\/p>\n <\/p>\n Performing data cleaning with Pandas is also easy.<\/p>\n <\/p>\n There is so much to do with Pandas for Data Manipulation. Check out Bala Priya article on using Pandas for Data Manipulation<\/a> to learn further.<\/p>\n <\/p>\n <\/p>\n Polars<\/a> is a relatively new data manipulation Python library designed for the swift analysis of large datasets. Polars boast 30x performance gains compared to Pandas in several benchmark tests.<\/p>\n Polars is built on top of the Apache Arrow, so it\u2019s efficient for memory management of the large dataset and allows for parallel processing. It also optimize their data manipulation performance using lazy execution that delays and computational until it\u2019s necessary.<\/p>\n For the Polars installation, you can use the following code.<\/p>\n <\/p>\n Like Pandas, you can initiate the Polars DataFrame with the following code.<\/p>\n However, there are differences in how we use Polars to manipulate data. For example, here is how we select data with Polars.<\/p>\n <\/p>\n The API is considerably more complex than Pandas, but it\u2019s helpful if you require fast execution for large datasets. On the other hand, you would not get the benefit if the data size is small.<\/p>\n To know the details, you can refer to Josep Ferrer's article on how different Polars is are compared to Pandas<\/a>.<\/p>\n <\/p>\n <\/p>\n Vaex<\/a> is similar to Polars as the library is developed specifically for considerable dataset data manipulation. However, there are differences in the way they process the dataset. For example, Vaex utilize memory-mapping techniques, while Polars focus on a multi-threaded approach.<\/p>\n Vaex is optimally suitable for datasets that are way bigger than what Polars intended to use. While Polars is also for extensive dataset manipulation processing, the library is ideally on datasets that still fit into memory size. At the same time, Vaex would be great to use on datasets that exceed the memory.<\/p>\n For the Vaex installation, it\u2019s better to refer to their documentation<\/a>, as it could break your system if it\u2019s not done correctly.<\/p>\n <\/p>\n <\/p>\npip install numpy<\/code><\/pre>\n<\/div>\nimport numpy as np\r\n\r\na = np.array([1, 2, 3])\r\nb = np.array([4, 5, 6])\r\nc = a + b\r\nprint(c)<\/code><\/pre>\n<\/div>\n[5 7 9]<\/code><\/p>\ndata = np.array([1, 2, 3, 4, 5, 6, 7])\r\nmean = np.mean(data)\r\nmedian = np.median(data)\r\nstd_dev = np.std(data)\r\n\r\nprint(f\"The data mean:{mean}, median:{median} and standard deviation: {std_dev}\")<\/code><\/pre>\n<\/div>\nx = np.array([[1, 2], [3, 4]])\r\ny = np.array([[5, 6], [7, 8]])\r\ndot_product = np.dot(x, y)\r\n\r\nprint(dot_product)<\/code><\/pre>\n<\/div>\n[[19 22]
\n[43 50]]<\/code><\/p>\n2. Pandas<\/h2>\n
import numpy as np\r\nimport pandas as pd\r\n\r\nnp.random.seed(0)\r\nmonths = pd.date_range(start='2023-01-01', periods=12, freq='M')\r\nsales = np.random.randint(10000, 50000, size=12)\r\ntransactions = np.random.randint(50, 200, size=12)\r\n\r\ndata = {\r\n'Month': months,\r\n'Sales': sales,\r\n'Transactions': transactions\r\n}\r\ndf = pd.DataFrame(data)\r\ndf.head()<\/code><\/pre>\n<\/div>\n
\n![\"Essential](\"https:\/\/www.kdnuggets.com\/wp-content\/uploads\/Wijaya_Essential_Python_Libraries_for_Data_Manipulation_2-300x218.png\")
<\/center>
\n <\/p>\ndf[df['Transactions'] <100]<\/code><\/pre>\n<\/div>\ntotal_sales = df['Sales'].sum() \r\naverage_transactions = df['Transactions'].mean() <\/code><\/pre>\n<\/div>\ndf = df.dropna() \r\ndf = df.fillna(df.mean()) <\/code><\/pre>\n<\/div>\n3. Polars<\/h2>\n
pip install polars <\/code><\/pre>\n<\/div>\nimport numpy as np\r\nimport polars as pl\r\n\r\nnp.random.seed(0) \r\nemployee_ids = np.arange(1, 101) \r\nages = np.random.randint(20, 60, size=100) \r\nsalaries = np.random.randint(30000, 100000, size=100) \r\n\r\ndf = pl.DataFrame({\r\n 'EmployeeID': employee_ids,\r\n 'Age': ages,\r\n 'Salary': salaries\r\n})\r\n\r\ndf.head()<\/code><\/pre>\n<\/div>\n
\n![\"Essential](\"https:\/\/www.kdnuggets.com\/wp-content\/uploads\/Wijaya_Essential_Python_Libraries_for_Data_Manipulation_3-232x300.png\")
<\/center>
\n <\/p>\ndf.filter(pl.col('Age') > 40)<\/code><\/pre>\n<\/div>\n4. Vaex<\/h2>\n
5. CuPy<\/h2>\n