Dropwizard Kotlin ‘Have I Been Pwned’ Password Resource

Dropwizard Kotlin ‘Have I Been Pwned’ Password Resource

This is actually Part 5 of the “Building a Kotlin Dropwizard Maven REST API from scratch” series but I have broken it out into it’s own post for two reasons. Firstly this part is only useful if you want your API to talk to another API, while very common, is not always the case. Secondly this could be useful on it’s own right, as you may want to ensure when a user is setting a password, that they aren’t using one that is known to potential hackers.

Have I Been Pwned (HIBP) is a resource that is available to tell you if your account or password has been involved in a known security data breach (scarily there are many). Being able to tap into this resource can be useful in any application that asks users to choose a password, so that you can be sure they have set a unique password (or one that isn’t known publicly).

To do that, we will call their password range API endpoint, it accepts the first five characters of a SHA-1 hashed password and returns a list of matching password in their SHA-1 hash format. Don’t worry if it isn’t immediately clear what this all means as I will break down each step of the way. The first thing we will need to do is configure a “client” so that we can connect and send requests to HIBP. To do this, we need to add another dependency to our pom.xml file.

<dependency>
    <groupId>io.dropwizard</groupId>
    <artifactId>dropwizard-client</artifactId>
    <version>${dropwizard.version}</version>
</dependency>

This will make the Javax http client available in your project as well as easy-to-configure Dropwizard configuration that we will address later. For now we want to create a clients folder/package and create a new PwnedClient.kt file. Inside we will define a class that takes a URL and a configured client to create a “target”, simply a preconfigured endpoint ready to accept requests. Then we will expose a function that will take five characters (trimming any excess) that it will pass to this endpoint and then return a map of matching hashes and the number of times the password has been exposed.

package com.mannanlive.dropwizardkotlinapp.clients

import javax.ws.rs.client.Client

class PwnedClient(url: String, client: Client) {
    private val target = client.target(url)

    fun getHashes(hashPrefix: String): Map<String, Int> {
        val response = target
                .path("range")
                .path(hashPrefix)
                .request()
                .get()
        val output = response.readEntity(String::class.java)
        return if (response.status == 200) {
            output.lines().map { line ->
                val sections = line.split(":")
                hashPrefix + sections[0] to sections[1].toInt()
            }.toMap()
        } else {
            throw RuntimeException("Error: ${response.status}, Body: $output")
        }
    }
}

Hopefully the above should be fairly readable, it is sending a request to: https://api.pwnedpasswords.com/range/ABCDE and getting a response with a list of password hashes and their count like:

003F292149C1BC6E313F59661DAA9BB7874:2
012A93CD6E5EE704FB1D2E3B238ED2D4A37:146
...

To do this, programatically the code is doing the following:
1. creating a request to GET: /range/{fiveCharacters}
e.g. /range/00000 to /range/FFFFF
2. getting the response from HIBP as a raw string value
3. if the response is not successful (200 OK) throw an exception
4. otherwise
a) turn the response to a list of strings, one for each line
b) splitting the line into two segments
segment 1 -> the remaining password hash (missing the first five characters)
segment 2 -> the number of times it has been pwned
c) turn the above two with a map, with the hash as the key (adding the original five characters back on the front) and the count as the value

But hang on, what is a SHA-1 password hash and how do we get one? It is an algorithm that turns an input value into a “message digest”. For example, the value “password” becomes “5BAA61E4C9B93F3F0682250B6CF8331B7EE68FD8” after being processed via the algorithm. The principle behind it is storing the raw password in plaintext is a really really bad idea, and storing a hashed value that you can check matches an input value is a lot safer. SHA-1 is considered a pretty weak hash by todays standards, but this is what HIBP requires and is why we are using it.

To create one in Kotlin, we need to create a src/main/kotlin/../services folder/package and create a PwnedService.kt class. We need a way to generate a SHA-1 hash, we can use a function written by Sam Clarke to do this. We need to call our newly created PwnedClient class above with this hash, we need to see if any of the results matches the input. Finally we need to return the results to the caller, with a value of 0 if no matches was found.

package com.mannanlive.dropwizardkotlinapp.services

import com.mannanlive.dropwizardkotlinapp.clients.PwnedClient
import java.security.MessageDigest

data class PwnedResult(val input: String, val timesPwned: Int)

class PwnedService(private val client: PwnedClient) {
    fun getNumberOfTimesPwned(password: String): PwnedResult {
        val shaHash = hashString("SHA-1", password)
        val firstFiveChars = shaHash.substring(0, 5)
        val potentialMatches = client.getHashesWithCache(firstFiveChars)
        val numberOfTimesPwned: Int = potentialMatches[shaHash] ?: 0
        return PwnedResult(password, numberOfTimesPwned)
    }

    private fun hashString(type: String, input: String): String {
        val HEX_CHARS = "0123456789ABCDEF"
        val bytes = MessageDigest
                .getInstance(type)
                .digest(input.toByteArray())
        val result = StringBuilder(bytes.size * 2)

        bytes.forEach {
            val i = it.toInt()
            result.append(HEX_CHARS[i shr 4 and 0x0f])
            result.append(HEX_CHARS[i and 0x0f])
        }

        return result.toString()
    }
}

You can see above on line 6, that we are using Kotlin’s data class to store the hash and the count in a simple object. On line 8 we create our service with a client as a required constructor parameter. On line 12 you can see that we use the hash to retrieve the value from the result map, and if null, use a value a 0 using an Elvis operator.

Now to expose this functionality to a consumer, we will need to write another REST resource. In the src/main/kotlin/../resources folder/package, we need to create a PwnedResource.kt file. It will accept requests to the /pwned endpoint and produce a result in application/json format. It will take a query parameter named password (defaulting to the literal value of “password” if not provided) and return the SHA-1 hash and the number of times it has been exposed, with a status of 200 OK if it has been exposed at least once, otherwise a 404 NOT FOUND if no matches were found.

package com.mannanlive.dropwizardkotlinapp.resources

import com.mannanlive.dropwizardkotlinapp.services.PwnedService
import javax.ws.rs.*
import javax.ws.rs.core.MediaType
import javax.ws.rs.core.Response

@Path("/pwned")
@Produces(MediaType.APPLICATION_JSON)
class PwnedResource(private val service: PwnedService) {
    @GET
    fun get(@QueryParam("password") @DefaultValue("password") password: String): Response {
        val result = service.getNumberOfTimesPwned(password)
        return if (result.timesPwned > 0) {
            Response.ok(result)
        } else {
            Response.status(404).entity(result)
        }.build()
    }
}

To the observant readers, you might have noticed we never configured the URL value or HTTP client, so let’s do that now and tie it all together. In our src/main/resource/app-config.yml file we created in Part 2, we need to add our configuration. We do this in configuration instead of code for a multitude of different reasons, such as wanting to target different endpoints for testing, development and production.

haveIBeenPwnedUrl: https://api.pwnedpasswords.com
httpClient:
  userAgent: dropwizard-kotlin-app

We need to set a User-Agent header with each request because it is part of the HIBP API is to always provide one. To use these values, we need to edit our configuration/AppConfig.kt file created earlier to know where to find these two new properties.

package com.mannanlive.dropwizardkotlinapp.configuration

import com.fasterxml.jackson.annotation.JsonProperty
import io.dropwizard.Configuration
import io.dropwizard.client.JerseyClientConfiguration

class AppConfig(
        @JsonProperty("testProp") val testProp: String,
        @JsonProperty("haveIBeenPwnedUrl") val haveIBeenPwnedUrl : String,
        @JsonProperty("httpClient") val httpClient : JerseyClientConfiguration
) : Configuration()

And finally, in our App.kt, in the run function, we need to create and register our new resource.

import io.dropwizard.client.JerseyClientBuilder
...
val client = JerseyClientBuilder(env).using(config.httpClient).build("httpClient")
val pwnedClient = PwnedClient(config.haveIBeenPwnedUrl, client)
env.jersey().register(PwnedResource(PwnedService(pwnedClient)))

Now if we run the application as we did in Part 4, we can send a request to our endpoint and get the state of a given password.

> curl -i http://localhost:8080/pwned?password=hackme
HTTP/1.1 200 OK
Date: Mon, 01 Apr 2019 12:16:53 GMT
Content-Type: application/json

{"input":"hackme","timesPwned":2339}

> curl -i http://localhost:8080/pwned?password=hackme1014
HTTP/1.1 404 Not Found
Date: Mon, 01 Apr 2019 12:19:13 GMT

{"input":"hackme1014","timesPwned":0}

Bonus Exercise: Tests

Testing should be optional, but in this exercise on calling REST APIs from Kotlin Dropwizard apps – it is. Below is an implementation of the above two scenarios, but in an automated fashion. This way if something introduces a breaking change, either your code or the HIBP endpoint, these tests should fail and alert us. In a future tutorial, I will show how to use Wiremock to stub out the HIBP service, allowing us to test difficult to simulate scenarios (timeouts, service down etc) and insulate ourselves from a flaky or unreliable service breaking our builds intermittently. Below create PwnedResourceIntegrationTest.kt in src/test/kotlin/../resources folder/package.

package com.mannanlive.dropwizardkotlinapp.resources

import com.mannanlive.dropwizardkotlinapp.App
import com.mannanlive.dropwizardkotlinapp.configuration.AppConfig
import io.dropwizard.testing.ResourceHelpers.resourceFilePath
import io.dropwizard.testing.junit.DropwizardAppRule
import org.assertj.core.api.Assertions.assertThat
import org.junit.ClassRule
import org.junit.Test
import javax.ws.rs.client.ClientBuilder
import javax.ws.rs.core.Response

data class ExpectedResult(
        val input: String? = null,
        val timesPwned: Int? = null)

class PwnedResourceIntegrationTest {
    companion object {
        @JvmField
        @ClassRule
        val RULE = DropwizardAppRule<AppConfig>(App::class.java,
                resourceFilePath("app-config.yml"))
    }

    @Test
    fun `can GET pwned password successfully`() {
        val response = sendRequest("hackme")
        assertThat(response.status).isEqualTo(200)
        val result = response.readEntity(ExpectedResult::class.java)
        assertThat(result.timesPwned).isGreaterThan(2000)
        assertThat(result.input).isEqualTo("hackme")
    }

    @Test
    fun `can GET unique password successfully`() {
        val response = sendRequest("hackme1014")
        assertThat(response.status).isEqualTo(404)
        val result = response.readEntity(ExpectedResult::class.java)
        assertThat(result.timesPwned).isEqualTo(0)
        assertThat(result.input).isEqualTo("hackme1014")
    }

    private fun sendRequest(inputPassword: String): Response = ClientBuilder.newClient()
            .target("http://localhost:${RULE.localPort}/pwned?password=$inputPassword")
            .request()
            .get()!!
}

Bonus: Caching

When calling another service, it is good etiquette and a great boost to performance to cache the request. This means subsequent requests don’t put any load on downstream systems. As HIBP does not get updated frequently it is a great opportunity to add caching. In our PwnedClient.kt, you can add the below import statements, property and method.

import com.google.common.cache.CacheBuilder
import com.google.common.cache.CacheLoader
import java.util.concurrent.TimeUnit
...
    private val cache = CacheBuilder.newBuilder()
            .expireAfterWrite(24, TimeUnit.HOURS)
            .build(CacheLoader.from(this::getHashes))

    fun getHashesWithCache(inputHash: String): Map<String, Int> = cache.get(inputHash)

Breaking this down, we are using Google’s Guava Cache Loader class, that wraps our getHashes function. It will auto-magically store the results of the call, so that any subsequent requests will return the previously loaded value. In addition, because we are using the first five characters of a SHA-1 hash we are also caching other values. For example both “hackme” and “7figures” hash starts with “CC959”, so a request for either actually caches the other (and over 100 other pwned passwords) which further helps reduce the load on HIBP and makes your application more responsive.

To use this, in the PwnedService, you can change client.getHashes call to be client.getHashesWithCache. If you have added the integration tests you can run them again and (hopefully) they should all continue to pass. You can also experiment with printing out a message in the getHashes function and see that it only prints once, regardless of how many requests it receives.

You may have noticed we have also configured the cache to automatically purge their entries after 24 hours so that the data will never be more than a day old. There are other configuration options available for you to experiment and play with if you like to experiment, but already it’s a really valuable tool when building micro-services.