Post Data Into Website Using Jsoup in Java

1. Overview

Jsoup is an open-source Java library used mainly for extracting data from HTML. It also allows you to manipulate and output HTML. It has a steady development line, great documentation, and a fluent and flexible API.

Apart from scraping webpages and manipulating HTML, it also provides a convenient API for making HTTP requests.

In this article, we’ll use Jsoup’s Connection API to post data to a website and test POST requests using the public dummy API on httpbin.org. 

2. Maven Dependency

To use the jsoup library, we need to add the following dependency to the pom.xml:

<dependency>
    <groupId>org.jsoup</groupId>
    <artifactId>jsoup</artifactId>
    <version>1.21.2</version>
</dependency>

We can find the latest version of the Jsoup library in the Maven Central repository.

3. Send a POST Request 

Jsoup’s Connection interface is an HTTP client that fetches content from the web and parses it into Documents. It’ll allow us to configure Jsoup to act like a client that can post data. Let’s take a look at the method below to understand this in detail:

public String sendPost(String url, Map<String, String> headers, String jsonPayload)
  throws Exception {
    Connection.Response response = Jsoup.connect(url)
      .headers(headers)
      .requestBody(jsonPayload)
      .method(Connection.Method.POST)
      .ignoreContentType(true)
      .execute();
    return response.body();
}

3.1. Preparing the Request

We’ll create a connection object using Jsoup.connect(). This will set up the request for the given url. This connection object will serve as a builder, allowing us to add more details before executing the request.

In general, headers in HTTP requests serve as metadata. They carry information such as authentication details, content types, versions, and more. We can set these headers using the .headers(headers). 

We can provide our data using .requestBody(jsonPayload). Jsoup sends the raw JSON exactly as we provide it, without altering the body. After that, we need to specify the HTTP method using .method(). By default, Jsoup assumes a GET request. We can override this behavior by passing the method as  Connection.Method.POST. 

3.2. Handling the Response

Jsoup usually works with HTML and expects HTML responses, whereas APIs often return a JSON response. The ignoreContentType() method of the Connection object is used to control how Jsoup handles the Content-Type response header.

By default, ignoreContentType is set to false. This means that Jsoup will check the response header’s Content-Type. If it’s not in a recognized text-based format like application/html, text/* or application/xml, it’ll throw an UnsupportedMimeTypeException.

Our server returns plain text or JSON, so we set ignoreContentType to true. This way, Jsoup won’t try to enforce an HTML-only response. It’ll allow us to work with JSON, XML, or any other data type returned by the API.

3.3. Executing the Call

Up to this point, everything we’ve done has been just configuration. We haven’t sent anything over the network yet. When we call .execute(), Jsoup actually fires the HTTP request.

Jsoup connects to the target server using the given URL. It attaches all the headers and payload we provided while constructing the request. Jsoup blocks the program until the server responds, ensuring we receive the response before proceeding. Once the server replies, Jsoup captures the status code, the response body, and headers. All this information will be stored in a Connection object.

Now, we can call response.body() to get the raw body of the response as a string. We can then parse, log, or convert this string into objects.

4. Posting Data on HttpBin

Httpbin is a simple and free HTTP request and response service. Let’s use the method sendPost() discussed above to post some data on it:

public void givenJSONData_whenUsingHttpBin_thenPostContent() throws Exception {
    Map<String, String> headersMap = new HashMap<>();
    headersMap.put("Content-Type", "application/json");
    String payload = "{ \"name\": \"Joe\", \"role\": \"Tester\" }";
    String response = client.sendPost("https://httpbin.org/post", headersMap, payload);

    assertNotNull(response);
    assertTrue(response.contains("Joe"));
    assertTrue(response.contains("Tester"));
}

In the above method, we first create a map of HTTP headers. We’re setting the Content-Type to application/json, which tells the server that we’re sending the JSON data. Next, we’re creating a JSON string that represents the data we want to send in the request body. Then we call the sendPost() method discussed above to send the request.

The response includes the same JSON we sent because httpbin.org returns whatever data we post. Next, we can run assertions to make sure everything is working as expected.

5. Conclusion

In this tutorial, we explored how we can send a POST request using Jsoup.

We explored how to use Jsoup as a lightweight HTTP client, apart from a HTML parser. We also understood about the importance of ignoreContentType when dealing with APIs that return JSON data. Our test on httpbin confirmed that Jsoup is able to send the request and return the server response as a raw string.

Jsoup may not replace a full-featured HTTP client for complex use cases, but it works well for sending JSON payloads and integrating lightweight API calls into our applications.