Integration testing of HTTP(S) communication with Java

Sometimes you want to be 100% sure your HTTP(s) client works exactly like it promises. That means you'll need tests. But what kind of? Just one real-life example: file downloading. How are you going to simulate connection lost, noisy network, broken data and all that weird network incidents? It could be done, partially, with the standalone HTTP servers. But they should be configured and have 100% uptime to use them in the continuous integration. Yes, we have mocks. They are unbeatable warriors of business logic testing. But, remember, we want to check everything, in fact, we just want to go through the real Client App <---> Network <---> Server App path.
That what I'm going to talk about in this article: we'll create a test which starts embedded HTTP(S) server (configured with our own HTTP handler) and runs the network-enabled code.

A regular note before we start - all the code below, packed in a ready-to-launch maven project, is waiting for you on the GitHub.
And one more remark: I give much credit to the company I'm working in - GridGain. A considerable part of the code for this project has been taken (or inspired by) the company's private repository.

Ok, suppose we have a network-capable utility class which can download files by the URL:

public class NetUtils {
  /**
   * Downloads resource by URL into file.
   *
   * @param url URL to download.
   * @param file File where downloaded resource should be stored.
   * @return File where downloaded resource should be stored.
   * @throws IOException If error occurred.
   */
  public static File downloadUrl(URL url, File file) throws IOException {
    ...
  }
}

Yep, the primary target to test is NetUtils.downloadUrl() method. And I'm going to cover three test cases: file download from HTTP url, HTTPS and local file system one.
As you might guess, I don't want to create another one super tiny lightweight open source HTTP(S) server. Instead of this I'll use the one embedded in Oracle JDK - classes HttpServer and HttpsServer. To make HTTP-related testing smoother I'll introduce the wrapper over these JDK classes - EmbeddedHttpServer with pretty simple functionality: start, stop, handle HTTP(S) request / check some predicates while request handling.

public class EmbeddedHttpServer {
  /**
   * Creates and starts embedded HTTP server.
   *
   * @return Started HTTP server instance.
   */
  public static EmbeddedHttpServer startHttpServer() throws Exception {
    ...
  }

  /**
   * Creates and starts embedded HTTPS server.
   *
   * @return Started HTTPS server instance.
   */
  public static EmbeddedHttpServer startHttpsServer() throws Exception {
    ...
  }

  /**
   * Configures server with test-specific HTTP handler.
   *
   * @return Configured HTTP(s) server.
   */
  public EmbeddedHttpServer withHandler(HttpHandler handler) {
    ...
  }

  /**
   * Stops server by closing the listening socket and disallowing any new exchanges
   * from being processed.
   *
   * @param delay Maximum time in seconds to wait until exchanges have finished.
   */
  public void stop(int delay) {
    ...
  }
}

While EmbeddedHttpServer class definition is tremendously simple the method withHandler() deserves special attention. In fact, it's a cornerstone of a test - you check and handle HTTP(S) requests with your custom HttpHandler implementations. To test the NetUtils.downloadUrl() we'll need the FileDownloadingHandler.

/**
 * The class represents a server handler triggered on incoming request.
 * The handler checks that a request is a HTTP GET and that url path is the expected one.
 * If all checks are passed it writes pre-configured file content to the HTTP response body.
 */
public class FileDownloadingHandler implements HttpHandler {
  /** URL path. */
  private final String urlPath;

  /** File to be downloaded. */
  private final File downloadFile;

  /**
   * Creates and configures FileDownloadingHandler.
   *
   * @param urlPath Url path on which a future GET request is going to be executed.
   * @param fileToBeDownloaded File to be written into the HTTP response.
   */
  public FileDownloadingHandler(String urlPath, File fileToBeDownloaded) {
    checkNotNull(urlPath);
    checkArgument(fileToBeDownloaded.exists());

    this.urlPath = urlPath;
    this.downloadFile = fileToBeDownloaded;
  }

  /**
   * Handles HTTP requests: checks that a request is a HTTP GET and that url path is the expected one.
   * If all checks are passed it writes pre-configured file content to the HTTP response body.
   *
   * @param exchange Wrapper above the HTTP request and response.
   */
  @Override public void handle(HttpExchange exchange) throws IOException {
    checkArgument("GET".equalsIgnoreCase(exchange.getRequestMethod()));

    // Check that a request has come to expected URL path.
    checkState(urlPath.equals(exchange.getRequestURI().toString()));

    exchange.getResponseHeaders().set("Content-Type", "application/octet-stream");
    exchange.sendResponseHeaders(200, 0);

    OutputStream resBody = exchange.getResponseBody();
    try {
      resBody.write(FileUtils.readFileToByteArray(downloadFile));
    }
    finally {
      resBody.close();
    }
  }
}

And now the tests go.

public class NetUtilsTest {
  private File downloadedFile;
  private File fileToBeDownloaded;

  @Before
  public void before() throws Exception {
    fileToBeDownloaded = getFileToBeDownloaded();
  }

  @After
  public void after() {
    if (!downloadedFile.delete())
      fail();
  }

  @Test
  public void downloadUrlFromHttp() throws Exception {
    EmbeddedHttpServer srv = null;

    try {
      String urlPath = "/testDownloadUrl/";

      srv = EmbeddedHttpServer.startHttpServer().withHandler(new FileDownloadingHandler(urlPath, fileToBeDownloaded));

      downloadedFile = new File(System.getProperty("java.io.tmpdir") + File.separator + "url-http.file");

      downloadedFile = NetUtils.downloadUrl(new URL(srv.getBaseUrl() + urlPath), downloadedFile);

      assertTrue(FileUtils.contentEquals(fileToBeDownloaded, downloadedFile));
    } finally {
      if (srv != null)
        srv.stop(1);
    }
  }

  @Test
  public void downloadUrlFromHttps() throws Exception {
    EmbeddedHttpServer srv = null;

    try {
      String urlPath = "/testDownloadUrl/";

      srv = EmbeddedHttpServer.startHttpsServer().withHandler(new FileDownloadingHandler(urlPath, fileToBeDownloaded));

      downloadedFile = new File(System.getProperty("java.io.tmpdir") + File.separator + "url-http.file");

      downloadedFile = NetUtils.downloadUrl(new URL(srv.getBaseUrl() + urlPath), downloadedFile);

      assertTrue(FileUtils.contentEquals(fileToBeDownloaded, downloadedFile));
    } finally {
      if (srv != null)
        srv.stop(1);
    }
  }

  @Test
  public void downloadUrlFromLocalFileSystem() throws Exception {
    downloadedFile = new File(System.getProperty("java.io.tmpdir") + File.separator + "url-http.file");

    downloadedFile = NetUtils.downloadUrl(fileToBeDownloaded.toURI().toURL(), downloadedFile);

    assertTrue(FileUtils.contentEquals(fileToBeDownloaded, downloadedFile));
  }

  private File getFileToBeDownloaded() throws URISyntaxException {
    return new File(this.getClass().getResource("/download.me").toURI());
  }
}

As you see, in the downloadUrlFromHttp and downloadUrlFromHttps tests we go through the same steps:

1. Start embedded HTTP(S) server, tell it to use FileDownloadingHandler which will not only serve the requests but also check that incoming requests are made to the given url path.
2. Call NetUtils.downloadUrl to download the given url into a temporary file.
3. Check the downloaded file and the original one have the same content.

Obviously there is no need to launch any kind of server for the downloadUrlFromLocalFileSystem test.

So what we got here? Pluggable, ready to-use-in-tests embedded HTTP(S) server which can be be started/stopped right in a test case. If you look at the EmbeddedHttpServer sources you'll find another small feature - the server automatically discovers free network port to bind.