{"id":4024,"date":"2020-12-16T11:24:55","date_gmt":"2020-12-16T05:54:55","guid":{"rendered":"https:\/\/copyassignment.com\/?p=4024"},"modified":"2022-06-18T16:56:59","modified_gmt":"2022-06-18T11:26:59","slug":"python-web-scraping-using-python-selenium","status":"publish","type":"post","link":"https:\/\/copyassignment.com\/python-web-scraping-using-python-selenium\/","title":{"rendered":"Python Web scraping using Python Selenium"},"content":{"rendered":"\n<hr class=\"wp-block-separator has-css-opacity\"\/>\n\n\n\n<h2 class=\"wp-block-heading\">Definition:<\/h2>\n\n\n\n<hr class=\"wp-block-separator has-css-opacity\"\/>\n\n\n\n<p>Before understanding Python web scraping, let&#8217;s first have a look at what is webscrapping.  In simple terms, web scraping is the automated process of extracting something from a website. This automation helps to process the HTML of a web page and manipulate it to convert the web page into a different format, copy it into a database, and many different activities.  Web Scraping can be used to access the World Wide Web through HyperText Transfer Protocol. Additionally, this can also be done via Web Browsers. Web Scraping can be done manually also. In Python webscraping, we use Python Selenium for web scraping purposes. Web Scraping is sometimes done for illegal purposes too. If a business relies on its content and business model, scraping can sometimes make them a huge financial loss.<\/p>\n\n\n\n<hr class=\"wp-block-separator has-css-opacity\"\/>\n\n\n\n<h2 class=\"wp-block-heading\"><strong>Techniques of Web Scraping:<\/strong><\/h2>\n\n\n\n<hr class=\"wp-block-separator has-css-opacity\"\/>\n\n\n\n<p>Accordingly, there are many techniques for web scraping. For example, some of them are listed below&#8211;<\/p>\n\n\n\n<ul class=\"myfontsize wp-block-list\"><li><strong>Human Copy Paste<\/strong>: Human Copy Paste is the simplest method of web scraping. This method indulges the manual copy and pasting of data from a web page into a spreadsheet. This can sometimes be the best method of web scraping too. This is because some websites have very thick protection towards automation.<\/li><li><strong>Text Pattern Matching<\/strong>: This simple, however compelling web scraping method is based on the UNIX grep command or Regular Expression or known as RegEx, of programming languages like Perl and Python.<\/li><li><strong>HTTP Programming<\/strong>: Using socket programming, we can post HTTP requests to a remote webserver to retrieve the static or dynamic web pages.<\/li><li><strong>HTML Parsing<\/strong>: As we know, many websites have pages generated dynamically from a source like a database. Here, the page will have a template. Likewise, a program that detects such templates is called a wrapper. So, the information from a specific script can be extracted. This is mainly used in data mining.<\/li><li><strong>DOM Parsing<\/strong>: As we know that Internet Explorer and Mozilla Firefox are some full-fledged web browsers. By embedding these web browsers, programs can extract content generated by client-side scripts. XPath can be used in this method.<\/li><\/ul>\n\n\n\n<hr class=\"wp-block-separator has-css-opacity\"\/>\n\n\n\n<h2 class=\"wp-block-heading\"><strong>Some uses of Web Scraping<\/strong> :<\/h2>\n\n\n\n<hr class=\"wp-block-separator has-css-opacity\"\/>\n\n\n\n<p>Simultaneously, there are many uses of web scraping. For instance, some of them are listed below&#8211;<\/p>\n\n\n\n<ul class=\"myfontsize wp-block-list\"><li>Web Data Integrating <\/li><li>Data Mining<\/li><li>Detecting change in a website<\/li><li>Online presence checking<\/li><li>Weather Data Monitoring<\/li><\/ul>\n\n\n\n<hr class=\"wp-block-separator has-css-opacity\"\/>\n\n\n\n<h2 class=\"wp-block-heading\"><strong>There are 6 types of testing<\/strong>:<\/h2>\n\n\n\n<hr class=\"wp-block-separator has-css-opacity\"\/>\n\n\n\n<ul class=\"myfontsize wp-block-list\"><li>Acceptance testing<\/li><li>Functional testing<\/li><li>Performance testing<\/li><li>Regression testing<\/li><li>Test-driven development (TDD)<\/li><li>Behavior-driven development (BDD)<\/li><\/ul>\n\n\n\n<hr class=\"wp-block-separator has-css-opacity\"\/>\n\n\n\n<script async=\"\" src=\"https:\/\/pagead2.googlesyndication.com\/pagead\/js\/adsbygoogle.js\"><\/script>\n<ins class=\"adsbygoogle\" style=\"display:block; text-align:center;\" data-ad-layout=\"in-article\" data-ad-format=\"fluid\" data-ad-client=\"ca-pub-9886351916045880\" data-ad-slot=\"2002566052\"><\/ins>\n<script>\n     (adsbygoogle = window.adsbygoogle || []).push({});\n<\/script>\n\n\n\n<hr class=\"wp-block-separator has-css-opacity\"\/>\n\n\n\n<h2 class=\"wp-block-heading\"><strong>Languages that support Selenium:<\/strong><\/h2>\n\n\n\n<hr class=\"wp-block-separator has-css-opacity\"\/>\n\n\n\n<ul class=\"myfontsize wp-block-list\"><li>Java<\/li><li>Python<\/li><li>C#<\/li><li>Ruby<\/li><li>JavaScript<\/li><li>Kotlin<\/li><\/ul>\n\n\n\n<hr class=\"wp-block-separator has-css-opacity\"\/>\n\n\n\n<h2 class=\"wp-block-heading\"><strong>Introduction to Selenium<\/strong><\/h2>\n\n\n\n<hr class=\"wp-block-separator has-css-opacity\"\/>\n\n\n\n<p>Selenium is a tool that helps us to automate web browsers. Selenium is used for testing web applications. However, it can do much more stuff. It opens up the web browser and does things a normal human being would do. Some of those things include clicking buttons, searching for information, filling the input field even though it is a handy tool. If you scrape a website frequently and for malicious purposes, your IP address may be banned as it is against most of the websites&#8217; terms and services.<\/p>\n\n\n\n<hr class=\"wp-block-separator has-css-opacity\"\/>\n\n\n\n<h3 class=\"wp-block-heading\"><strong>A. Installation<\/strong><\/h3>\n\n\n\n<hr class=\"wp-block-separator has-css-opacity\"\/>\n\n\n\n<h4 class=\"wp-block-heading\"><em>Requirements:<\/em> <\/h4>\n\n\n\n<p>Python 2 should be 2.7.9 or higher, OR Python 3 should be 3.4.0 or higher. Pip should also be installed.<\/p>\n\n\n\n<p>Run the below command in terminal.<\/p>\n\n\n\n<div class=\"maindiv\"><div class=\"codediv\"><strong><pre style=\"color:#cf2e2e\">pip install selenium\n<\/pre><\/strong>\n<\/div><\/div>\n\n\n\n<hr class=\"wp-block-separator has-css-opacity\"\/>\n\n\n\n<h4 class=\"wp-block-heading\"><em>Drivers: <\/em><\/h4>\n\n\n\n<hr class=\"wp-block-separator has-css-opacity\"\/>\n\n\n\n<p>Browsers require selenium WebDrivers to an interface. Different browsers have different drivers. To illustrate, some of the most popular drivers are given below.<\/p>\n\n\n\n<figure class=\"wp-block-table\"><table><tbody><tr><td>Chrome<\/td><td><a rel=\"noreferrer noopener\" href=\"https:\/\/sites.google.com\/a\/chromium.org\/chromedriver\/\" target=\"_blank\">Chrome Drive<\/a><a href=\"https:\/\/sites.google.com\/a\/chromium.org\/chromedriver\/\" target=\"_blank\" rel=\"noreferrer noopener\">r<\/a><\/td><\/tr><tr><td>Edge<\/td><td><a href=\"https:\/\/developer.microsoft.com\/en-us\/microsoft-edge\/tools\/webdriver\/#downloads\" data-type=\"URL\" data-id=\"https:\/\/developer.microsoft.com\/en-us\/microsoft-edge\/tools\/webdriver\/#downloads\">Edge Driver<\/a><\/td><\/tr><tr><td>Firefox<\/td><td><a href=\"https:\/\/github.com\/mozilla\/geckodriver\/releases\" data-type=\"URL\" data-id=\"https:\/\/github.com\/mozilla\/geckodriver\/releases\">Gecko Driver<\/a><\/td><\/tr><tr><td>Opera<\/td><td><a href=\"https:\/\/github.com\/operasoftware\/operachromiumdriver\/releases\" data-type=\"URL\" data-id=\"https:\/\/github.com\/operasoftware\/operachromiumdriver\/releases\">Opera Driver<\/a><\/td><\/tr><tr><td>Safari<\/td><td>Safari WedDriver is built-in<\/td><\/tr><\/tbody><\/table><\/figure>\n\n\n\n<hr class=\"wp-block-separator has-css-opacity\"\/>\n\n\n\n<h3 class=\"wp-block-heading\">B. <strong>Getting Started<\/strong><\/h3>\n\n\n\n<hr class=\"wp-block-separator has-css-opacity\"\/>\n\n\n\n<h4 class=\"wp-block-heading\"><em>Simple Use<\/em> :<\/h4>\n\n\n\n<p>As you install web driver and pip module, you can start using selenium with python as below:<\/p>\n\n\n\n<div style=\"height: 250px;margin-bottom: 50px\" class=\"wp-block-simple-code-block-ace\"><div style=\"cursor:pointer\" class=\"copy-simple-code-block\"><span class=\"dashicon dashicons dashicons-admin-page\"><\/span><\/div><pre class=\"wp-block-simple-code-block-ace\" data-mode=\"python\" data-theme=\"xcode\" data-fontsize=\"14\" data-lines=\"Infinity\" data-showlines=\"false\" data-copy=\"true\">from selenium import webdriver\nfrom selenium.webriver.common.keys import Keys\n# Driver specification and executable path\ndriver = webdriver.Chrome(executable_path=\"PATH TO WEBDRIVER\")\n# Redirects to the specified website\ndriver.get(\"violet-cat-415996.hostingersite.com\")\n#specifies the input element in the website\nsearch_element = driver.find_element_by_id(\"is-search-input-0\")\ntext = \"Turtle\" # sends the keys to the input field\nsearch.send_keys(\"Turtle\")<\/pre><\/div>\n\n\n\n<hr class=\"wp-block-separator has-css-opacity\"\/>\n\n\n\n<p>The above code will browse into <a href=\"https:\/\/copyassignment.com\/\" data-type=\"URL\" data-id=\"https:\/\/copyassignment.com\/\">https:\/\/copyassignment.com\/<\/a>, and it will search Turtle in the search box. You can replace the value of the &#8220;text&#8221; variable.<\/p>\n\n\n\n<hr class=\"wp-block-separator has-css-opacity\"\/>\n\n\n\n<h4 class=\"wp-block-heading\">Writing tests with selenium:<\/h4>\n\n\n\n<p>As we know, selenium is mostly used for writing test cases. However, selenium itself does not come with testing tools. For instance, we should use python modules like pytest or nose.<\/p>\n\n\n\n<p>On the contrary, we will be using the unittest module. The following code for Python Web Scraping will check the functionality of search on python.org.<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code><strong>import<\/strong> <strong>unittest<\/strong>\n<strong>from<\/strong> <strong>selenium<\/strong> <strong>import<\/strong> webdriver\n<strong>from<\/strong> <strong>selenium.webdriver.common.keys<\/strong> <strong>import<\/strong> Keys\n\n<strong>class<\/strong> <strong>PythonOrgSearch<\/strong>(unittest.TestCase):\n\n    <strong>def<\/strong> setUp(self):\n        self.driver = webdriver.Firefox()\n\n    <strong>def<\/strong> test_search_in_python_org(self):\n        driver = self.driver\n        driver.get(\"http:\/\/www.python.org\")\n        self.assertIn(\"Python\", driver.title)\n        elem = driver.find_element_by_name(\"q\")\n        elem.send_keys(\"pycon\")\n        elem.send_keys(Keys.RETURN)\n        <strong>assert<\/strong> \"No results found.\" <strong>not<\/strong> <strong>in<\/strong> driver.page_source\n\n\n    <strong>def<\/strong> tearDown(self):\n        self.driver.close()\n\n<strong>if<\/strong> __name__ == \"__main__\":\n    unittest.main()<\/code><\/pre>\n\n\n\n<p> Now, if you want to run the above code from shell, follow these steps.<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>python test_python_org_search.py\n.\n----------------------------------------------------------------------\nRan 1 test <strong>in<\/strong> 15.566s\n\nOK<\/code><\/pre>\n\n\n\n<p>We can clearly see that the test was successful. In addition, you can even use this in Jupyter or iPython.<\/p>\n\n\n\n<hr class=\"wp-block-separator has-css-opacity\"\/>\n\n\n\n<script async=\"\" src=\"https:\/\/pagead2.googlesyndication.com\/pagead\/js\/adsbygoogle.js\"><\/script>\n<ins class=\"adsbygoogle\" style=\"display:block; text-align:center;\" data-ad-layout=\"in-article\" data-ad-format=\"fluid\" data-ad-client=\"ca-pub-9886351916045880\" data-ad-slot=\"2002566052\"><\/ins>\n<script>\n     (adsbygoogle = window.adsbygoogle || []).push({});\n<\/script>\n\n\n\n<hr class=\"wp-block-separator has-css-opacity\"\/>\n\n\n\n<h4 class=\"wp-block-heading\">Selenium Remote WebDriver for Python Selenium:<\/h4>\n\n\n\n<p>If you want to use the remote web driver, then you should have the selenium server running. After the server is running, you can use some of the following examples.<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code><strong>from<\/strong> <strong>selenium<\/strong> <strong>import<\/strong> webdriver\n<strong>from<\/strong> <strong>selenium.webdriver.common.desired_capabilities<\/strong> <strong>import<\/strong> DesiredCapabilities\n\ndriver = webdriver.Remote(\n   command_executor='http:\/\/127.0.0.1:4444\/wd\/hub',\n   desired_capabilities=DesiredCapabilities.CHROME)\n\ndriver = webdriver.Remote(\n   command_executor='http:\/\/127.0.0.1:4444\/wd\/hub',\n   desired_capabilities=DesiredCapabilities.OPERA)\n\ndriver = webdriver.Remote(\n   command_executor='http:\/\/127.0.0.1:4444\/wd\/hub',\n   desired_capabilities=DesiredCapabilities.HTMLUNITWITHJS)<\/code><\/pre>\n\n\n\n<p>As we know that the expected capability is a dictionary. Instead, you can specify the values as:<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>driver = webdriver.Remote(\n   command_executor='http:\/\/127.0.0.1:4444\/wd\/hub',\n   desired_capabilities={'browserName': 'htmlunit',\n                         'version': '2',\n                        'javascriptEnabled': <strong>True<\/strong>})<\/code><\/pre>\n\n\n\n<h3 class=\"wp-block-heading\">C. <strong><em>Navigation<\/em><\/strong><\/h3>\n\n\n\n<p><em>Interaction with HTML<\/em>:<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>&lt;input type=\"text\" name=\"inp\" id=\"fill\" class=\"sam\"&gt;<\/code><\/pre>\n\n\n\n<p>In the above case, if you want to navigate the HTML element, then you can use any of the following.<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>scr = driver.find_element_by_name(\"inp\")\nscr = driver.find_element_by_id(\"fill\")\nscr = driver.find_element_by_class(\"sam\")\nscr = driver.find_element_by_xpath(\"\/\/input&#91;@id='fill']\")\nscr = driver.find_element_by_css_selector(\"input#fill\")<\/code><\/pre>\n\n\n\n<p><em><span style=\"text-decoration: underline\">NOTE<\/span><\/em>: The ID of an element should be unique. This means that no other element in the webpage should have the same ID. If there is a duplicate ID, selenium will recognize the first one.<\/p>\n\n\n\n<p><em>Sending Text to input field<\/em>:<\/p>\n\n\n\n<p>Furthermore, for sending a text to the input field, you should import one more thing. That is likely,<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>from selenium.webdriver.common.keys import Keys<\/code><\/pre>\n\n\n\n<p>Now, you can use the above import in the following manner.<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>scr.send_keys(\"Sample Text\")<\/code><\/pre>\n\n\n\n<p>Similarly, the table below shows the code for entering the keys on the keyboard.<\/p>\n\n\n\n<h4 class=\"wp-block-heading\">Key Codes:<\/h4>\n\n\n\n<table border=\"1\" width=\"100%\">\n<tbody>\n<tr>\n<td>ADD<\/td>\n<td>ALT<\/td>\n<td>ARROW_DOWN<\/td>\n<\/tr>\n<tr>\n<td>ARROW_LEFT<\/td>\n<td>ARROW_RIGHT<\/td>\n<td>ARROW_UP<\/td>\n<\/tr>\n<tr>\n<td>BACKSPACE<\/td>\n<td>BACK_SPACE<\/td>\n<td>CANCEL<\/td>\n<\/tr>\n<tr>\n<td>CLEAR<\/td>\n<td>COMMAND<\/td>\n<td>CONTROL<\/td>\n<\/tr>\n<tr>\n<td>DECIMAL<\/td>\n<td>DELETE<\/td>\n<td>DIVIDE<\/td>\n<\/tr>\n<tr>\n<td>DOWN<\/td>\n<td>END<\/td>\n<td>ENTER<\/td>\n<\/tr>\n<tr>\n<td>EQUALS<\/td>\n<td>ESCAPE<\/td>\n<td>F1<\/td>\n<\/tr>\n<tr>\n<td>F10<\/td>\n<td>F11<\/td>\n<td>F12<\/td>\n<\/tr>\n<tr>\n<td>F2<\/td>\n<td>F3<\/td>\n<td>F4<\/td>\n<\/tr>\n<tr>\n<td>F5<\/td>\n<td>F6<\/td>\n<td>F7<\/td>\n<\/tr>\n<tr>\n<td>F8<\/td>\n<td>F9<\/td>\n<td>HELP<\/td>\n<\/tr>\n<tr>\n<td>HOME<\/td>\n<td>INSERT<\/td>\n<td>LEFT<\/td>\n<\/tr>\n<tr>\n<td>LEFT_ALT<\/td>\n<td>LEFT_CONTROL<\/td>\n<td>LEFT_SHIFT<\/td>\n<\/tr>\n<tr>\n<td>META<\/td>\n<td>MULTIPLY<\/td>\n<td>NULL<\/td>\n<\/tr>\n<tr>\n<td>NUMPAD0<\/td>\n<td>NUMPAD1<\/td>\n<td>NUMPAD2<\/td>\n<\/tr>\n<tr>\n<td>NUMPAD3<\/td>\n<td>NUMPAD4<\/td>\n<td>NUMPAD5<\/td>\n<\/tr>\n<tr>\n<td>NUMPAD6<\/td>\n<td>NUMPAD7<\/td>\n<td>NUMPAD8<\/td>\n<\/tr>\n<tr>\n<td>NUMPAD9<\/td>\n<td>PAGE_DOWN<\/td>\n<td>PAGE_UP<\/td>\n<\/tr>\n<tr>\n<td>PAUSE<\/td>\n<td>RETURN<\/td>\n<td>RIGHT<\/td>\n<\/tr>\n<tr>\n<td>SEMICOLON<\/td>\n<td>SEPARATOR<\/td>\n<td>SHIFT<\/td>\n<\/tr>\n<tr>\n<td>SPACE<\/td>\n<td>SUBTRACT<\/td>\n<td>TAB<\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n\n\n\n<hr class=\"wp-block-separator has-css-opacity\"\/>\n\n\n\n<script async=\"\" src=\"https:\/\/pagead2.googlesyndication.com\/pagead\/js\/adsbygoogle.js\"><\/script>\n<ins class=\"adsbygoogle\" style=\"display:block; text-align:center;\" data-ad-layout=\"in-article\" data-ad-format=\"fluid\" data-ad-client=\"ca-pub-9886351916045880\" data-ad-slot=\"2002566052\"><\/ins>\n<script>\n     (adsbygoogle = window.adsbygoogle || []).push({});\n<\/script>\n\n\n\n<hr class=\"wp-block-separator has-css-opacity\"\/>\n\n\n\n<p><em>Drag and Drop:<\/em><\/p>\n\n\n\n<p>You can use this feature by either moving the specified element or on to another element. For using the drag and drop feature you can use the following code of Python Selenium.<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>from selenium.webdriver import ActionChains\n\nelement = driver.find_element_by_name(\"source\")\ntarget = driver.find_element_by_name(\"target\")\n\naction_chains = ActionChains(driver)\naction_chains.drag_and_drop(element, target).perform()<\/code><\/pre>\n\n\n\n<p><em>Moving from frame to frame:<\/em><\/p>\n\n\n\n<p>Nowadays, most websites have frames. So, there is a different method to switch between frames on a web page. The method used is &#8220;switch_to_window.&#8221; For example, le the HTML code be like<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>&lt;a href=\"somewhere.html\" target=\"windowName\"&gt;Click here to open a new window&lt;\/a&gt;<\/code><\/pre>\n\n\n\n<p>Then, the python code will be<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>driver.switch_to_window(\"windowName\")\n<\/code><\/pre>\n\n\n\n<p><em>Popups:<\/em><\/p>\n\n\n\n<p>Selenium has built-in support to handle popup boxes. There is an example of a popup handling below:<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>alert = driver.switch_to.alert<\/code><\/pre>\n\n\n\n<p><em>Cookies:<\/em><\/p>\n\n\n\n<p>Additionally, selenium also has cookie uses. You can understand the cookies in selenium from the code below. <\/p>\n\n\n\n<pre class=\"wp-block-code\"><code># Navigating to the domain\ndriver.get(\"http:\/\/www.example.com\")\n\n# Setting the cookie\ncookie = {\u2018name\u2019 : \u2018aakriti\u2019, \u2018value\u2019 : \u2018girl\u2019}\ndriver.add_cookie(cookie)\n\n# This will output all the available cookies\ndriver.get_cookies()<\/code><\/pre>\n\n\n\n<p><em><span style=\"text-decoration: underline\">Fun Fact<\/span><\/em>: We know that Selenium with Python is a popular way of testing web apps, but selenium supports many different languages. Some of them are C#, Java, C++, etc.<\/p>\n\n\n\n<h3 class=\"wp-block-heading\">D. <em>Locating<\/em><\/h3>\n\n\n\n<p>There are many ways to locate elements on a web page in Python Web scraping using Python Selenium. You can use it according to the environment you want. The methods to locate elements on a web page are:<\/p>\n\n\n\n<ul class=\"myfontsize wp-block-list\"><li><strong>find_element_by_id<\/strong><\/li><li><strong>find_element_by_name<\/strong><\/li><li><strong>find_element_by_xpath<\/strong><\/li><li><strong>find_element_by_link_text<\/strong><\/li><li><strong>find_element_by_partial_link_text<\/strong><\/li><li><strong>find_element_by_tag_name<\/strong><\/li><li><strong>find_element_by_class_name<\/strong><\/li><li><strong>find_element_by_css_selector<\/strong><\/li><\/ul>\n\n\n\n<p>If you want to find many elements on a page then you can use the following methods.<\/p>\n\n\n\n<ul class=\"myfontsize wp-block-list\"><li><strong>find_elements_by_name<\/strong><\/li><li><strong>find_elements_by_xpath<\/strong><\/li><li><strong>find_elements_by_link_text<\/strong><\/li><li><strong>find_elements_by_partial_link_text<\/strong><\/li><li><strong>find_elements_by_tag_name<\/strong><\/li><li><strong>find_elements_by_class_name<\/strong><\/li><li><strong>find_elements_by_css_selector<\/strong><\/li><\/ul>\n\n\n\n<p><em><span style=\"text-decoration: underline\">Note<\/span><\/em>: The above methods will return a list.<\/p>\n\n\n\n<p>These are used in the example below.<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>from selenium.webdriver.common.by import By\n\ndriver.find_element(By.XPATH, '\/\/input&#91;text()=\"Some text\"]')\ndriver.find_elements(By.XPATH, '\/\/input')<\/code><\/pre>\n\n\n\n<p>Moreover, there are some attributes available for <em>By<\/em> class. They are listed below.<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>ID = \"id\"\nXPATH = \"xpath\"\nLINK_TEXT = \"link text\"\nPARTIAL_LINK_TEXT = \"partial link text\"\nNAME = \"name\"\nTAG_NAME = \"tag name\"\nCLASS_NAME = \"class name\"\nCSS_SELECTOR = \"css selector\"<\/code><\/pre>\n\n\n\n<h4 class=\"wp-block-heading\">1. Locating an element by ID<\/h4>\n\n\n\n<p>We should use this when we know the id of an element. If the id provided is wrong, there will be an error called <em>NoSuchElementException. Additionally, if we use this strategy,<\/em> then selenium will recognize the first element with the ID.<\/p>\n\n\n\n<p>To illustrate let the source of the page be like:<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>&lt;html&gt;\n &lt;body&gt;\n  &lt;form id=\"loginForm\"&gt;\n   &lt;input name=\"username\" type=\"text\"&gt;\n   &lt;input name=\"password\" type=\"password\"&gt;\n   &lt;input name=\"continue\" type=\"submit\" value=\"Login\"&gt;\n  &lt;\/form&gt;\n &lt;\/body&gt;\n&lt;html&gt;<\/code><\/pre>\n\n\n\n<p>Here, the form element can be located like:<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>form = driver.find_element_by_id('loginForm')<\/code><\/pre>\n\n\n\n<h4 class=\"wp-block-heading\">2. Locating an element by Name<\/h4>\n\n\n\n<p>We should use this when we know the name of an element. If the name provided is wrong, there will be an error called NoSuchElementException. Additionally, if we use this strategy, then selenium will recognize the first element with the name. This is the same as above.<\/p>\n\n\n\n<p>For instance, let the source of a page be:<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>&lt;html&gt;\n &lt;body&gt;\n  &lt;form id=\"loginForm\"&gt;\n   &lt;input name=\"username\" type=\"text\"&gt;\n   &lt;input name=\"password\" type=\"password\"&gt;\n   &lt;input name=\"continue\" type=\"submit\" value=\"Login\"&gt;\n   &lt;input name=\"continue\" type=\"button\" value=\"Clear\"&gt;\n  &lt;\/form&gt;\n&lt;\/body&gt;\n&lt;html&gt;<\/code><\/pre>\n\n\n\n<p>Therefore, the python code to locate the username and password in Python Web scraping using Python Selenium will be as follows. <\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>username = driver.find_element_by_name('username')\npassword = driver.find_element_by_name('password')<\/code><\/pre>\n\n\n\n<p>Moreover, to locate login button, we use the code<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code><strong>continue<\/strong> = driver.find_element_by_name('continue')<\/code><\/pre>\n\n\n\n<p><em><span style=\"text-decoration: underline\">Note<\/span><\/em>: Selenium will locate the login button because it occurs before clear.<\/p>\n\n\n\n<h4 class=\"wp-block-heading\">3. Locating with XPath<\/h4>\n\n\n\n<p>XPath is a language that is used to locate nodes in XML documents. XPath supports the simple locating methods by id or name attributes and extends them by opening up all sorts of new possibilities, such as locating the third checkbox on the page.<br><br>One of the main reasons for using XPath is when you don\u2019t have a suitable id or name attribute for the element you wish to locate.<br><br>You can use XPath to either locate the element in absolute terms or relative to an element that does have an id or name attribute.<br><br>XPath locators can also be used to specify elements via attributes other than id and name.<br><br>By finding a nearby element with an id or name attribute, you can locate your target element based on the relationship.<\/p>\n\n\n\n<p>Let the source of a page be:<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>&lt;html&gt;\n &lt;body&gt;\n  &lt;form id=\"loginForm\"&gt;\n   &lt;input name=\"username\" type=\"text\" \/&gt;\n   &lt;input name=\"password\" type=\"password\" \/&gt;\n   &lt;input name=\"continue\" type=\"submit\" value=\"Login\" \/&gt;\n   &lt;input name=\"continue\" type=\"button\" value=\"Clear\" \/&gt;\n  &lt;\/form&gt;\n&lt;\/body&gt;\n&lt;html&gt;<\/code><\/pre>\n\n\n\n<p>So, the code to locate the form element will be<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>login_form = driver.find_element_by_xpath(\"\/html\/body\/form&#91;1]\")\nlogin_form = driver.find_element_by_xpath(\"\/\/form&#91;1]\")\nlogin_form = driver.find_element_by_xpath(\"\/\/form&#91;@id='loginForm']\")<\/code><\/pre>\n\n\n\n<p>Consequently, the codes to locate username element could be:<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>username = driver.find_element_by_xpath(\"\/\/form&#91;input\/@name='username']\")\nusername = driver.find_element_by_xpath(\"\/\/form&#91;@id='loginForm']\/input&#91;1]\")\nusername = driver.find_element_by_xpath(\"\/\/input&#91;@name='username']\")<\/code><\/pre>\n\n\n\n<p> Likewise, the clear element can be locate as follows.<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>clear_button = driver.find_element_by_xpath(\"\/\/input&#91;@name='continue']&#91;@type='button']\")\nclear_button = driver.find_element_by_xpath(\"\/\/form&#91;@id='loginForm']\/input&#91;4]\")<\/code><\/pre>\n\n\n\n<hr class=\"wp-block-separator has-css-opacity\"\/>\n\n\n\n<script async=\"\" src=\"https:\/\/pagead2.googlesyndication.com\/pagead\/js\/adsbygoogle.js\"><\/script>\n<ins class=\"adsbygoogle\" style=\"display:block; text-align:center;\" data-ad-layout=\"in-article\" data-ad-format=\"fluid\" data-ad-client=\"ca-pub-9886351916045880\" data-ad-slot=\"2002566052\"><\/ins>\n<script>\n     (adsbygoogle = window.adsbygoogle || []).push({});\n<\/script>\n\n\n\n<hr class=\"wp-block-separator has-css-opacity\"\/>\n\n\n\n<h4 class=\"wp-block-heading\">4. Locating Hyperlinks by Link Text<\/h4>\n\n\n\n<p>You should use this strategy if the link text is inside the anchor tag.<br>When we use this method, we get the link text of the element specified. If there is no element like the specified one, there will be an error called <em>NoSuchElementException<\/em>.<\/p>\n\n\n\n<p>For instance, let the source of the page be like this.<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>&lt;html&gt;\n &lt;body&gt;\n  &lt;p&gt;Are you sure you want to do this?&lt;\/p&gt;\n  &lt;a href=\"next.html\"&gt;Next&lt;\/a&gt;\n  &lt;a href=\"previous.html\"&gt;Back&lt;\/a&gt;\n&lt;\/body&gt;\n&lt;html&gt;<\/code><\/pre>\n\n\n\n<p>Subsequently, the next.html and previous.html can be located as.<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>continue_link = driver.find_element_by_link_text('Next')\ncontinue_link = driver.find_element_by_partial_link_text('Back')<\/code><\/pre>\n\n\n\n<h4 class=\"wp-block-heading\">5. Locating Elements by Tag Name<\/h4>\n\n\n\n<p>You should use this strategy if the desired content is to be located by tag name. When we use this method, we get the first element with the matching class name attribute. If there is no element like the specified one, there will be an error called <em>NoSuchElementException<\/em>.<\/p>\n\n\n\n<p>For example, let the source be:<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>&lt;html&gt;\n &lt;body&gt;\n  &lt;p class=\"cont\"&gt;lorem ipsum.&lt;\/p&gt;\n&lt;\/body&gt;\n&lt;html&gt;<\/code><\/pre>\n\n\n\n<p>Here, the &#8220;p&#8221; element can be located as<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>loc = driver.find_element_by_class_name('cont')<\/code><\/pre>\n\n\n\n<h4 class=\"wp-block-heading\"><strong>6. Localting Elements by CSS Selector<\/strong><\/h4>\n\n\n\n<p>You can use this method when you want to locate with CSS selector syntax. Likewise, if there is no matching element, there will be an error. The error, as always, is <em>NoSuchElementException<\/em>.<\/p>\n\n\n\n<p>To illustrate, let the html code be:<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>&lt;html&gt;\n &lt;body&gt;\n  &lt;p class=\"cont\"&gt;Lorem ipsum.&lt;\/p&gt;\n&lt;\/body&gt;\n&lt;html&gt;<\/code><\/pre>\n\n\n\n<p>Likely, the code to locate &#8220;p&#8221; tag will be:<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>content = driver.find_element_by_css_selector('p.cont')<\/code><\/pre>\n\n\n\n<h4 class=\"wp-block-heading\">E. <strong><em>Wait<\/em><\/strong><\/h4>\n\n\n\n<p>Nowadays, many websites are using the AJAX technique. In this technique, the elements of the page load at different times. In contrast, this will create a problem many times. When a locate function runs, but the element is not loaded, there will be an error. The error is called ElementNotVisibleException. Now, to solve this issue, we have a feature in selenium called waits. Nonetheless, we can add some intervals between locating elements or performing actions on them using this feature. <\/p>\n\n\n\n<p>The Selenium WebDriver provides two types of waits are Implicit Waits and Explicit Waits. Similarly, they are explained below.<\/p>\n\n\n\n<h5 class=\"wp-block-heading\">1. <strong>Explicit Waits<\/strong><\/h5>\n\n\n\n<p>In this wait, we make the program wait for some time until the specified condition is fulfilled. The highest wait is time.sleep(), which can be used by importing the python&#8217;s built-in module called &#8220;time.&#8221; To be precise, one of the ways to do this is using WebDriverWait and ExpectedCondition together.<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code><strong>from<\/strong> <strong>selenium<\/strong> <strong>import<\/strong> webdriver\n<strong>from<\/strong> <strong>selenium.webdriver.common.by<\/strong> <strong>import<\/strong> By\n<strong>from<\/strong> <strong>selenium.webdriver.support.ui<\/strong> <strong>import<\/strong> WebDriverWait\n<strong>from<\/strong> <strong>selenium.webdriver.support<\/strong> <strong>import<\/strong> expected_conditions <strong>as<\/strong> EC\n\ndriver = webdriver.Firefox()\ndriver.get(\"http:\/\/somedomain\/url_that_delays_loading\")\n<strong>try<\/strong>:\n    element = WebDriverWait(driver, 10).until(\n        EC.presence_of_element_located((By.ID, \"myDynamicElement\"))\n    )\n<strong>finally<\/strong>:\n    driver.quit()<\/code><\/pre>\n\n\n\n<p>If we run the above python code, it will wait for a maximum of 10 seconds for finding the matching element to be found. If there is no element found until the given time, then TimeoutException will occur. The output will be either true or null if success and fail, respectively. <\/p>\n\n\n\n<h6 class=\"wp-block-heading\">A. Conditions Expected <\/h6>\n\n\n\n<p>While automating the web in Python Web scraping using Python Selenium there are some common conditions that are expected. Some of them are listed below.<\/p>\n\n\n\n<ul class=\"myfontsize wp-block-list\"><li>title_is.<\/li><li>title_contains.<\/li><li>presence_of_element_located.<\/li><li>visibility_of_element_located.<\/li><li>visibility_of.<\/li><li>presence_of_all_elements_located.<\/li><li>text_to_be_present_in_element.<\/li><li>text_to_be_present_in_element_value.<\/li><li>frame_to_be_available_and_switch_to_it.<\/li><li>invisibility_of_element_located.<\/li><li>element_to_be_clickable.<\/li><li>staleness_of.<\/li><li>element_to_be_selected.<\/li><li>element_located_to_be_selected.<\/li><li>element_selection_state_to_be.<\/li><li>element_located_selection_state_to_be.<\/li><li>alert_is_present.<\/li><\/ul>\n\n\n\n<pre class=\"wp-block-code\"><code><strong>from<\/strong> <strong>selenium.webdriver.support<\/strong> <strong>import<\/strong> expected_conditions <strong>as<\/strong> EC\n\nwait = WebDriverWait(driver, 10)\nelement = wait.until(EC.element_to_be_clickable((By.ID, 'someid')))<\/code><\/pre>\n\n\n\n<h6 class=\"wp-block-heading\">B. Custom Wait Conditions<\/h6>\n\n\n\n<pre class=\"wp-block-code\"><code><strong>class<\/strong> <strong>element_has_css_class<\/strong>(object):\n  <em>\"\"\"An expectation for checking that an element has a particular css class.<\/em>\n\n<em>  locator - used to find the element<\/em>\n<em>  returns the WebElement once it has the particular css class<\/em>\n<em>  \"\"\"<\/em>\n  <strong>def<\/strong> __init__(self, locator, css_class):\n    self.locator = locator\n    self.css_class = css_class\n\n  <strong>def<\/strong> __call__(self, driver):\n    element = driver.find_element(*self.locator)   <em># Finding the referenced element<\/em>\n    <strong>if<\/strong> self.css_class <strong>in<\/strong> element.get_attribute(\"class\"):\n        <strong>return<\/strong> element\n    <strong>else<\/strong>:\n        <strong>return<\/strong> <strong>False<\/strong>\n\n<em># Wait until an element with id='myNewInput' has class 'myCSSClass'<\/em>\nwait = WebDriverWait(driver, 10)\nelement = wait.until(element_has_css_class((By.ID, 'myNewInput'), \"myCSSClass\"))<\/code><\/pre>\n\n\n\n<h5 class=\"wp-block-heading\"><strong>2. Implicit Waits<\/strong><\/h5>\n\n\n\n<p>Implicit Waits are very rarely used in Selenium. This tells the web driver to poll the DOM for a certain time when the specified element is not immediately available.<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code><strong>from<\/strong> <strong>selenium<\/strong> <strong>import<\/strong> webdriver\n\ndriver = webdriver.Firefox()\ndriver.implicitly_wait(10) <em># seconds<\/em>\ndriver.get(\"http:\/\/somedomain\/url_that_delays_loading\")\nmyDynamicElement = driver.find_element_by_id(\"myDynamicElement\")<\/code><\/pre>\n\n\n\n<h4 class=\"wp-block-heading\"><strong>F. <em>Page Objects<\/em><\/strong><\/h4>\n\n\n\n<p>Finally, we will learn about page object design patterns. In general, The area of a web page where the test is interacting is called page objects.<\/p>\n\n\n\n<p><strong>Some benefits of using page objects patterns are:<\/strong><\/p>\n\n\n\n<ul class=\"myfontsize wp-block-list\"><li>Code that can be used in many places can be written.<\/li><li>Code that is written many times can be reduced.<\/li><li>Similarly, If there are changes in the user interface, the code of only one place should be replaced.<\/li><\/ul>\n\n\n\n<p><strong>1. Test Case<\/strong><\/p>\n\n\n\n<p>Here is a code which searches if a text is present in the given domain.<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code><strong>import<\/strong> <strong>unittest<\/strong>\n<strong>from<\/strong> <strong>selenium<\/strong> <strong>import<\/strong> webdriver\n<strong>import<\/strong> <strong>page<\/strong>\n\n<strong>class<\/strong> <strong>CopyAssignmentsSearch<\/strong>(unittest.TestCase):\n       <strong>def<\/strong> setUp(self):\n        self.driver = webdriver.Firefox()\n        self.driver.get(\"https:\/\/copyassignment.com\/\")\n\n    <strong>def<\/strong> test_search_in_python_org(self):\n        <em>\"\"\"<\/em>\n<em>        Tests<\/em> violet-cat-415996.hostingersite.com<em> search feature. Searches for the word \"Anaconda\" then verified that some results show up.<\/em>\n<em>        Note that it does not look for any particular text in search results page. This test verifies that<\/em>\n<em>        the results were not empty.<\/em>\n<em>        \"\"\"<\/em>\n\n        <em>#Load the main page. In this case the home page of <\/em>violet-cat-415996.hostingersite.com<em>.<\/em>\n        main_page = page.MainPage(self.driver)\n        <em>#Checks if the word \"Anaconda\" is in title<\/em>\n        <strong>assert<\/strong> main_page.is_title_matches(), \"violet-cat-415996.hostingersite.com title doesn't match.\"\n        <em>#Sets the text of search textbox to \"Anaconda\"<\/em>\n        main_page.search_text_element = \"pycon\"\n        main_page.click_go_button()\n        search_results_page = page.SearchResultsPage(self.driver)\n        <em>#Verifies that the results page is not empty<\/em>\n        <strong>assert<\/strong> search_results_page.is_results_found(), \"No results found.\"\n\n    <strong>def<\/strong> tearDown(self):\n        self.driver.close()\n\n<strong>if<\/strong> __name__ == \"__main__\":\n    unittest.main()<\/code><\/pre>\n\n\n\n<p><strong>2. Page Object Classes<\/strong><\/p>\n\n\n\n<p>The POP helps in creating a separation between test code and technical implementation.<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code><strong>from<\/strong> <strong>element<\/strong> <strong>import<\/strong> BasePageElement\n<strong>from<\/strong> <strong>locators<\/strong> <strong>import<\/strong> MainPageLocators\n\n<strong>class<\/strong> <strong>SearchTextElement<\/strong>(BasePageElement):\n    <em>\"\"\"This class gets the search text from the specified locator\"\"\"<\/em>\n\n    <em>#The locator for search box where search string is entered<\/em>\n    locator = 'q'\n\n\n<strong>class<\/strong> <strong>BasePage<\/strong>(object):\n    <em>\"\"\"Base class to initialize the base page that will be called from all pages\"\"\"<\/em>\n\n    <strong>def<\/strong> __init__(self, driver):\n        self.driver = driver\n\n\n<strong>class<\/strong> <strong>MainPage<\/strong>(BasePage):\n    <em>\"\"\"Home page action methods come here. I.e. Python.org\"\"\"<\/em>\n\n    <em>#Likewise, Declares a variable that will contain the retrieved text<\/em>\n    search_text_element = SearchTextElement()\n\n    <strong>def<\/strong> is_title_matches(self):\n        <em>\"\"\"Verifies that the hardcoded text \"Python\" appears in page title\"\"\"<\/em>\n        <strong>return<\/strong> \"Python\" <strong>in<\/strong> self.driver.title\n\n    <strong>def<\/strong> click_go_button(self):\n        <em>\"\"\"Triggers the search\"\"\"<\/em>\n        element = self.driver.find_element(*MainPageLocators.GO_BUTTON)\n        element.click()\n\n\n<strong>class<\/strong> <strong>SearchResultsPage<\/strong>(BasePage):\n    <em>\"\"\"Search results page action methods come here\"\"\"<\/em>\n\n    <strong>def<\/strong> is_results_found(self):\n        <em>#Finally,  Probably should search for this text in the specific page<\/em>\n        <em># element, but as for now it works fine<\/em>\n        <strong>return<\/strong> \"No results found.\" <strong>not<\/strong> <strong>in<\/strong> self.driver.page_source<\/code><\/pre>\n\n\n\n<p><strong>3. Page Elements<\/strong><\/p>\n\n\n\n<pre class=\"wp-block-code\"><code><strong>from<\/strong> <strong>selenium.webdriver.support.ui<\/strong> <strong>import<\/strong> WebDriverWait\n\n\n<strong>class<\/strong> <strong>BasePageElement<\/strong>(object):\n    <em>\"\"\"Base page class that is initialized on every page object class.\"\"\"<\/em>\n\n    <strong>def<\/strong> __set__(self, obj, value):\n        <em>\"\"\"Sets the text to the value supplied\"\"\"<\/em>\n        driver = obj.driver\n        WebDriverWait(driver, 100).until(\n            <strong>lambda<\/strong> driver: driver.find_element_by_name(self.locator))\n        driver.find_element_by_name(self.locator).clear()\n        driver.find_element_by_name(self.locator).send_keys(value)\n\n    <strong>def<\/strong> __get__(self, obj, owner):\n        <em>\"\"\"Gets the text of the specified object\"\"\"<\/em>\n        driver = obj.driver\n        WebDriverWait(driver, 100).until(\n            <strong>lambda<\/strong> driver: driver.find_element_by_name(self.locator))\n        element = driver.find_element_by_name(self.locator)\n        <strong>return<\/strong> element.get_attribute(\"value\")<\/code><\/pre>\n\n\n\n<hr class=\"wp-block-separator has-css-opacity\"\/>\n\n\n\n<script async=\"\" src=\"https:\/\/pagead2.googlesyndication.com\/pagead\/js\/adsbygoogle.js\"><\/script>\n<ins class=\"adsbygoogle\" style=\"display:block; text-align:center;\" data-ad-layout=\"in-article\" data-ad-format=\"fluid\" data-ad-client=\"ca-pub-9886351916045880\" data-ad-slot=\"2002566052\"><\/ins>\n<script>\n     (adsbygoogle = window.adsbygoogle || []).push({});\n<\/script>\n\n\n\n<hr class=\"wp-block-separator has-css-opacity\"\/>\n\n\n\n<p><strong>4. Locators<\/strong><\/p>\n\n\n\n<p>In this tradition, the locators of the page are kept in the same class where they are being used.<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code><strong>from<\/strong> <strong>selenium.webdriver.common.by<\/strong> <strong>import<\/strong> By\n\n<strong>class<\/strong> <strong>MainPageLocators<\/strong>(object):\n    <em>\"\"\"A class for main page locators. All main page locators should come here\"\"\"<\/em>\n    GO_BUTTON = (By.ID, 'submit')\n\n<strong>class<\/strong> <strong>SearchResultsPageLocators<\/strong>(object):\n    <em>\"\"\"A class for search results locators. All search results locators should come here\"\"\"<\/em>\n    <strong>pass<\/strong><\/code><\/pre>\n\n\n\n<hr class=\"wp-block-separator has-css-opacity\"\/>\n\n\n\n<h3 class=\"wp-block-heading\">Other Tools Like Selenium:<\/h3>\n\n\n\n<hr class=\"wp-block-separator has-css-opacity\"\/>\n\n\n\n<ul class=\"myfontsize wp-block-list\"><li>Katalon Studio<\/li><li>Subject7<\/li><li>Screenster<\/li><li>TestCraft<\/li><li>Endtest<\/li><li>Browsersync<\/li><li>Protractor<\/li><li>CasperJS<\/li><\/ul>\n\n\n\n<hr class=\"wp-block-separator has-css-opacity\"\/>\n\n\n\n<h3 class=\"wp-block-heading\">Some Other Python Web Scraping Tools:<\/h3>\n\n\n\n<hr class=\"wp-block-separator has-css-opacity\"\/>\n\n\n\n<ul class=\"myfontsize wp-block-list\"><li>Beautiful Soup<\/li><li>LXML<\/li><li>MechanicalSoup<\/li><li>Python Requests<\/li><li>Scrapy<\/li><li>urllib<\/li><\/ul>\n\n\n\n<hr class=\"wp-block-separator has-css-opacity\"\/>\n\n\n\n<p>This is a complete python web scraping tutorial and I hope you have learned enough from it as I have tried all the basics which I know by myself. Still, the article of design is left which will be done soon.<\/p>\n\n\n\n<p>Comment if you have any queries or found something wrong in the article or on our website.<\/p>\n\n\n\n<p>Thanks for Reading<\/p>\n\n\n\n<p class=\"has-medium-font-size\"><strong>Keep Learning<\/strong><\/p>\n\n\n\n<hr class=\"wp-block-separator has-css-opacity\"\/>\n\n\n\n<p style=\"font-size:22px\"><strong>Also Read:<\/strong><\/p>\n\n\n\n<hr class=\"wp-block-separator has-css-opacity\"\/>\n\n\n\n<p><a href=\"https:\/\/copyassignment.com\/download-chrome-driver-for-selenium\/\">Download Chrome Driver for Selenium<\/a><\/p>\n\n\n\n<p><a href=\"https:\/\/copyassignment.com\/automate-facebook-login-using-python-selenium\/\">Automate Facebook Login Using Python Selenium<\/a><\/p>\n\n\n\n<p><a href=\"https:\/\/copyassignment.com\/jarvis-and-google-assistant-voice-assistant-using-python\/\">Jarvis and Google Assistant || Voice Assistant Using Python<\/a><\/p>\n\n\n\n<p><a href=\"https:\/\/copyassignment.com\/crack-any-password-using-python\/\">Crack Any Password Using Python<\/a><\/p>\n\n\n\n<p><a href=\"https:\/\/copyassignment.com\/get-wifi-passwords-with-python\/\">Get WiFi Passwords With Python<\/a><\/p>\n\n\n\n<p><a href=\"https:\/\/copyassignment.com\/gui-application-to-check-wifi-password-in-python\/\">GUI Application To See wifi password in Python<\/a><\/p>\n\n\n\n<hr class=\"wp-block-separator has-css-opacity\"\/>\n","protected":false},"excerpt":{"rendered":"<p>Definition: Before understanding Python web scraping, let&#8217;s first have a look at what is webscrapping. In simple terms, web scraping is the automated process of&#8230;<\/p>\n","protected":false},"author":62,"featured_media":12835,"comment_status":"closed","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[22,1403],"tags":[1086,1085,1084,1087],"class_list":["post-4024","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-allcategorites","category-python-projects","tag-python-selenium","tag-python-web-scraping","tag-python-web-scraping-using-python-selenium","tag-selenium","wpcat-22-id","wpcat-1403-id"],"_links":{"self":[{"href":"https:\/\/copyassignment.com\/wp-json\/wp\/v2\/posts\/4024","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/copyassignment.com\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/copyassignment.com\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/copyassignment.com\/wp-json\/wp\/v2\/users\/62"}],"replies":[{"embeddable":true,"href":"https:\/\/copyassignment.com\/wp-json\/wp\/v2\/comments?post=4024"}],"version-history":[{"count":0,"href":"https:\/\/copyassignment.com\/wp-json\/wp\/v2\/posts\/4024\/revisions"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/copyassignment.com\/wp-json\/wp\/v2\/media\/12835"}],"wp:attachment":[{"href":"https:\/\/copyassignment.com\/wp-json\/wp\/v2\/media?parent=4024"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/copyassignment.com\/wp-json\/wp\/v2\/categories?post=4024"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/copyassignment.com\/wp-json\/wp\/v2\/tags?post=4024"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}