Written on December 8, 2014.

Tags: spring boot, kotlin

Table of contents

• What to expect from a dev framework for micro-services?
  • Kotlin
  • Spring Boot
• Example Webapp with Spring Boot in Kotlin
  • Architecture of a REST web app
  • REST endpoint
  • Configuration
  • Health check
  • Lazy initialization of loggers
  • Monitoring
• Conclusion

What to expect from a dev framework for micro-services?

In a start-up, the ability to deliver quickly makes the difference between surviving or disappearing. This is obviously not only tied to the technical part of the product development¹ but contribute to it. If you are in a SaaS model and you go for a micro-service architecture where each service is responsible of a functional aspect of your product, being able to produce, deploy, operate and maintain (new) services is key.

To do so, you need to define a framework – and to some extent, a process –with a number of key points I will try to enumerate. The more you standardize, the better it is. In this post, I will focus on the dev part with some considerations to help with the build and deployment but mostly let this for a future post.

So, from my point of view, the ops requirements are the following:

• Configuration: it should provide a convenient way to manage configuration. Here I would make a distinction between configuration that can be tuned by ops and the one that can be tuned by devs. For the former, it should be externalized so that it can be changed without a build. As an example, you can think of an AWS pair of keys or a S3 bucket name that your app might need to work. For the later, there is no need to externalize them and they can be in the war. Here you can think of length of some internal queue or any feature related parameter. The reason to make this distinction is to limit the number of required external parameters and facilitate the deployment.
• Health check: every service of your application should have a health check. This is useful to figure with component as failed and also to help load balancers to send requests to working servers. But a service can fail for multiple reasons, so having a precise information which tell why a particular service cannot work properly is essential.
• Monitoring: services need to be monitored for many reasons, from the early detection of failure to the proper scaling of your infrastructure but also to help figure out what will be the next bottleneck that need to be addressed as you grow. Exposing internal metrics from the service itself is a very effective way to get the metrics that have a meaning for your business. Rather than relying on system metrics to infer what is going on on your infra, if you have control of your code, make it expose the right metrics.
• Deployment: deployment should be as easy as possible cause you’ll have to automate it and do it so many time. Deploying your service by moving a single file is the goal.

And let looks at the dev requirements:

• Language targeting the JVM: let me rephrase this. You have to pick a runtime environment for your services. In many case the JVM is a good candidate but that depends on your constraints, what you/your devs are familiar with, what is typically used in your field (for existing open source libs)… Once you have it, the dev language has to target it.
• Constraining language: following Carmack’s words at QuakeCon 2013, everything that is syntaxically correct ends up in the codebase. So better choose a language that enforce good practices.
• Statically typed: there is no need facing runtime errors that the compiler could have pointed out.
• Type inference: there is no need to repeat yourself.
• Functional-style language: Though I’m not sure using a pure functional language like Haskell is practical for a micro-service architecture² Support for functional-style programming in the selected language is not just a nice to have. Many part of your services can be developed with pure functions that will be easy to test³ and refactor.
• Robust IDE support: though I loved coding with Emacs years ago, the benefits of using an IDE are numerous. To list few of them:
  • early error detection (think of null pointer detection in IntelliJ IDEA),
  • refactoring tools,
  • and obviously code navigation.

Kotlin

Kotlin is a language which target the JVM and Javascript developed by JetBrains. I won’t go too much in the details and refer you to the reference documentation for a complete description of the language. Still few points worth being said:

It is in the vein of the language offering functional-style programming in addition to classical imperative-style programming that target the JVM together with Ceylon and Scala.

Among its features, we can note its compiler has type inference, it supports extension function to add new functions to existing classes.

One of the cool features of Kotlin attempt to fix C. A. R. Hoare’s billion-dollar mistake by providing null safety in the language itself⁴. You probably know of @NotNull/@Nullable annotation for Java JetBrains is pushing⁵. With Kotlin you don’t have to annotate things. By default everything is assumed to be @NotNull but you can add a ? to the type to specify it can be null. In this case you wont be allow to call a method on it unless you check for the null case.

Lastly, there is a plugin for Intellij IDEA and maven so I can compile from both.

Spring Boot

Spring has a long history. Spring Boot is there to help quickly getting a standardized service based on Spring.

Out of the box, it already offers externalized configuration, REST endpoints exposing metrics and health checks, … But more importantly, XML is not required anymore and you can do everything from your code using annotations⁶.

Let’s see how it goes in practice.

Example Webapp with Spring Boot in Kotlin

The code of this example app is available on github: ssoudan/ktSpringTest. It is developed in Kotlin and make use of Spring Boot. I will detail some part of it in the following sections.

Architecture of a REST web app

I usually split the code of my REST services into the following packages:

• controllers where all the controllers are;
• services where the components of the layer with the business logic are;
• repositories where all the DAO are;
• configuration where Configuration classes are.

This permit to neatly layer your code as your app is layered: Your controllers will have Autowired to services, services will have Autowired to other service, repositories and configuration files. Repositories will have Autowired to configuration classes.

Spring Boot offers different stereotype annotations for them: RestController, Service, Repository, Configuration which can be used as markers to recognize the specific role of a Component.

Note, also, that the ComponentScan annotation on HelloApp class can be used to specify which package to scan for the annotated classes than must be converted to beans by Spring. This can be useful if you move some common code to another lib but still expect to be able to autowire beans defined in the lib from your app.

Now how do we create a REST endpoint?

REST endpoint

When you annotate you REST controller with RestController, it automatically tells that all the endpoints defined by methods of this bean expect their return value to be the body of the response⁷.

The rest is fairly simple, you annotate your methods to tell how to bind this method to a URL and HTTP method as in RequestMapping(array("/"), method = array(RequestMethod.GET)) where we tell that GET on ‘/’ should call this particular method. We can also annotate parameters of the method with RequestParam to bind HTTP request parameters to them as in:

RequestMapping(array("/"), method = array(RequestMethod.GET))
fun get(RequestParam("blah", required = false) blah : String?) : String
...

Some of the service that back a set of REST endpoints can require some configuration. Next section will show how to deal with that.

Configuration

The configuration is grouped in a class HelloConfiguration.kt where the fields are annotated with Value annotations both giving the name of the parameter (which can be defined in the application.properties, in an env variable, or via the command line when the app is launched. See Spring Boot - External config for a complete description).

Few tricks:

• If you want something to be defined or the app to fail to start, you can use Value(my.prop) without the ‘:’ and the default value. If ‘my.prop’ is not found, the app will fail to start.
• You can add more default property files as with PropertySource(value = array("classpath:/application.properties", "file:/etc/myapp.properties")) in HelloApp.kt so again you can split between the config which is specific to your service and the config which is global to a set of services working together.
• I recommend you to have two classes with the configuration, one in the service itself and one in a library you share will all the other services.

Configuration is obviously one of the aspects where the framework you put in place can help to operate the service in prod, but that’s not the only one. Being able to inspect the health of the running service is also essential.

Health check

We want a ‘/health’ endpoint that tell us if there is a problem and where it comes from. A convenient way is to have an endpoint which return a status code that tells us if there is a failure or not: 503 when it is not working, 200 when it is working – together with a JSON which tell which component has failed.

A Json like this would be ideal:

{
 "status":"UP",
 "serviceName":"ktSpringTest",
 "dependencies":[
    {"serviceName":"AnotherService","status":"UP"},
    {"serviceName":"HelloService","status":"UP","dependencies":[
        {"serviceName":"AnotherService","status":"UP"}]
    }
 ]
}

Spring Boot already provide a facility for a such health endpoint. Let’s tweak it to get what we want.

Spring Boot let us define our own AbstractHealthIndicator which get called when /health REST endpoint is called. We basically define our version which collect all the services implementing HealthCheckedService contract and iterate over to decide the status of the service and build the report.

Service
class ServiceHealthIndicator [Autowired] (val services: List<HealthCheckedService>,
                                          val configuration: HelloConfiguration) 
                                          : AbstractHealthIndicator() {

    fun updateStatus(status: HealthStatus, 
                     dependencies: List<HealthCheckedService.HealthInfo>): HealthStatus {
        var updatedStatus = status;
        dependencies.forEach { dep ->
            dep.updateStatus()
            if (dep.status == HealthStatus.DOWN) {
                updatedStatus = HealthStatus.DOWN;
            }
        }
        return updatedStatus
    }

    override fun doHealthCheck(p0: Health.Builder?) {

        if (p0 != null) {
            val dependencies = LinkedList<HealthCheckedService.HealthInfo>()
            services.forEach { i -> dependencies.add(i.healthCheck()) }

            val status = updateStatus(HealthStatus.UP, dependencies)

            p0.withDetail("dependencies", dependencies)
            p0.withDetail("serviceName", configuration.serviceName)

            if (status == HealthStatus.UP)
                p0.up()
            else
                p0.down()
        }
    }

}

HealthCheckedService is a contract we use on internal components that can report their status. It basically forces the inherited class to implement a healthCheck() method which returns a new HealthInfo object with the name and status of the services, plus its HealthChecked dependencies.

trait HealthCheckedService {

    enum class HealthStatus {
        UP
        DOWN
    }

    JsonInclude(Include.NON_EMPTY)
    data class HealthInfo(JsonIgnore val service: HealthCheckedService, var status: HealthStatus) {

        val dependencies: MutableList<HealthInfo> = LinkedList()

        val serviceName = service.javaClass.getSimpleName()

        fun addDependency(healthInfo: HealthInfo): HealthInfo {
            dependencies.add(healthInfo)
            return this
        }

        fun addDependency(service: HealthCheckedService): HealthInfo {
            dependencies.add(service.healthCheck())
            return this
        }

        fun updateStatus() {
            for (dep in dependencies) {
                dep.updateStatus()
                if (dep.status == HealthStatus.DOWN) {
                    this.status = HealthStatus.DOWN;
                }
            }
        }

    }

    fun healthCheck(): HealthInfo

}

We use updateStatus() to bubble up DOWN status in the dependence tree.

Now when we want to use it, we make our class implement HealthCheckService. For a service which do not depend on another service, we report its status as follow:

Service
public class AnotherService() : HealthCheckedService {
    override fun healthCheck(): HealthCheckedService.HealthInfo {
        return HealthInfo(this, HealthCheckedService.HealthStatus.UP)
    }

    public fun doSomething(): Observable<Long> {
        // blah
    }

}

While a service which rely on another service use addDependency() to build the tree:

Service
public class HelloService [Autowired] (val otherService : AnotherService ) : HealthCheckedService {
    override fun healthCheck(): HealthCheckedService.HealthInfo {
        return HealthInfo(this, HealthCheckedService.HealthStatus.UP).addDependency(otherService)
    }

    public fun getSomething(): Observable<Long> {
        // blah
    }

}

Note that your client libraries to another component can also report its ability to reach this component but cannot use the health endpoint for that or you risk to face an infinite loop of health check in case you have circular dependencies⁸.

Health endpoint gives a status of the app and its internal components. Sometime, logs are also a convenient way to figure out what is working properly and what is not. In the next section, we will see how we can use Kotlin to define and initialize a logger in a lazy manner.

Lazy initialization of loggers

Kotlin comes with few standard delegates. Delegates.lazy() is one of them and allow to define lazy initialized values.

Let’s use it for a logger. In HelloService.kt, we have:

val lazyLogger: Logger by Delegates.lazy {
    val logger = LoggerFactory.getLogger(javaClass<HelloService>())!!
    println("logger initialized!")
    logger
}

The two services HelloService and AnotherService which back (in cascade) the unique rest endpoint have respectively a call to lazyLogger.info("i'm here!"); and lazyLogger.info("and here!");

When we execute the server and call the REST endpoint, we see the following in the traces:

logger initialized!
42426 [qtp1605190078-20] INFO  e.s.k.services.HelloService - i'm here!
42426 [qtp1605190078-20] INFO  e.s.k.services.HelloService - and here!

The initialization is called just once, as expected!

Monitoring

By autowiring a MetricWriter to your component, you can register your own metrics that will get exposed via the ‘/metrics’ endpoint.

Service
public class HelloService [Autowired] (val metricWriter: MetricWriter, 
                                       val otherService : AnotherService) 
                                      : HealthCheckedService {

    override fun healthCheck(): HealthCheckedService.HealthInfo {
        return HealthInfo(this, HealthCheckedService.HealthStatus.UP)
                         .addDependency(otherService)
    }

    public fun getSomething(): Observable<Long> {
        lazyLogger.info("i'm here!");
        metricWriter.increment(Delta("counter.hello.service.call", 1));
        return otherService.doSomething().filter { it != null && 2L <= it } 
               as Observable<Long>
    }

}

Once this is done, the endpoint returns something similar to:

{
    "counter.hello.service.call":2, // <----
    "counter.status.200.health":1,
    "counter.status.200.root":3,
    "counter.status.404.error":1,
    "gauge.response.error":27.0,
    "gauge.response.health":83.0,
    "gauge.response.root":5.0,
    "mem":126976,
    "mem.free":87132,
    "processors":4,
    "uptime":43750777,
    "instance.uptime":43746263,
    "heap.committed":126976,
    "heap.init":131072,
    "heap.used":39843,
    "heap":1864192,
    "threads.peak":13,
    "threads.daemon":4,
    "threads":13,
    "classes":5405,
    "classes.loaded":5405,
    "classes.unloaded":0,
    "gc.ps_scavenge.count":21,
    "gc.ps_scavenge.time":71,
    "gc.ps_marksweep.count":2,
    "gc.ps_marksweep.time":213
}

I have not detailed all the tweaks you can do on your Spring Boot apps, you can find a list of ideas here and use this to define your own micro-service framework.

Conclusion

As a conclusion, I will just insist on few points:

• It is necessary to build a framework to standardize your production of micro-services if you go for this architecture: it will help you deliver quickly new features and also facilitate everyone’s tasks from development to deployment and operations.
• Spring Boot is a very interesting framework for that: it offers a lot of already existing features that you can customize to your needs.
• Kotlin is a promising language for software development but for now, still lack a significant traction. Would be curious to know if anyone is using Kotlin in prod yet?