Converting a jOOQ-Record into a vertx JsonObject

In vertx-jooq, there are a couple of ways to convert a jOOQ Record into a vertx JsonObject. First, checkout the straightforward way, which involves two steps: load the POJO using a generated DAO and then convert it with the toJson method.

It turns out however that on the way to a JsonObject, a lot of copying is going on. First the SELECT statement is mapped to a Record, then converted to a POJO and finally converted into a JsonObject. The code below is copied from DAOImpl class which is extended by every VertxDao:

If you’ve enabled the generation of interfaces in vertx-jooq (which is the default), then you could convert the Record directly into a JsonObject without converting to POJO first.

The major disadvantage of this solution is that you have to write the SELECT statement by yourself. On the other hand, you save the creation of the POJO, which is a plus. But what if we’re joining on another table so we cannot directly map into a generated Record? See this example:

Because the fetched Record is dynamic, we cannot simply call toJson on it. Instead we call the fetchOneMap method which “returns at most one resulting record as a name/value map.” Luckily, JsonObject has a constructor taking a Map<String,Object> which is exactly the same generic type returned (it even returns the same Map implementation that JsonObject uses under the hood: LinkedHashMap). It is noteworthy that this could even be done with pure jOOQ and vertx (leaving the execution aside). Some caveats though:

  1. No conversion happens. If you use a custom generator to generate the schema objects and have overwritten handleCustomTypeToJson / handleCustomTypeFromJson, you’re probably in bad shape. If one of the fields you’re fetching is one of those types, this conversion is bypassed.
  2. Because the constructor makes no checks and takes the map ‘as is’ without copying, you’re probably adding illegal JSON-types.
  3. The Map‘s key will have the name of the database column it is representing. This differs from the default-setting in vertx-jooq, which uses the name of the POJO’s member variable representing that column and thus would produce different JSON compared to the toJson method for the same Record/POJO. Starting with 2.3.5 however this can be changed so the keys are rendered with the same syntax.

Lessons learned

When simple CRUD is not enough, you have to fetch some joined Records and need to convert the results into a JsonObject, you should change the syntax how the JsonObject's keys are rendered in the POJO (3.). This is the first step in making both ways of generating JSON interoperable.

Things get trickier if you have to deal with conversion of custom JSON types. If your app/service is mostly reading data, you could handle conversion already on jOOQ-level so your converted type is already a valid JsonObject type. For example, if you’re using a SQL DateTime field “lastUpdated” on database level, just write a converter that formats the DateTime as a String. In turn, both the generated POJO and the fetchOneMap/fetchMaps methods will return a String for the “lastUpdated” entry and produce same results.

This can become a problem when those converted values need to be altered by the same app: a) it is more convenient and less error-prone to set a java.time.LocalDateTime object instead of dealing with Strings and b) some of the types may have special SQL-functions (e.g. dateAdd) which you cannot use any longer for that type.

Conclusion

Surprisingly there is no golden rule. First, I recommend to change the syntax of how JSON keys are rendered to be the same as the database column names (I will most likely change that behavior to the default in one of the next releases). This means you can use the fetchOneMap/fetchMaps methods most of the time to produce JSON. When dealing with custom types you should check how frequently those are updated by your app. If you’re facing a read-only app, write custom jOOQ converters for these types. If these types are altered quite often, you should stick to their actual type and handle the conversion into a proper JSON value on your own.

Advertisements

Hello async my old friend

Let me present you vertx-jooq-async: the world’s first fully async and type-safe SQL code generation and execution tool for VertX™. Async? Wasn’t it asynchronous before? It was, but what the code did was wrapping JDBC-calls using Vertx’ executeBlocking-method, which just shifts the blocking code from one event loop to another (see also).

1r9aly

With this release however, things have changed. jOOQ is still used for generating code and creating type-safe queries, but code is executed utilizing vertx-mysql-postgresql-client. That library in turn is based on a non-blocking driver for MySQL and Postgres databases, so this time it is really non-blocking.

Although some stuff is yet missing – you should go and check out the github-page. Now.

The Future Is Here

Getting asynchronous operations right is hard. And when you have to pass a result of one function to the next, it can only get worse. Since today, vertx-jooq’s API for instance only allowed asynchronous operations with a callback handler, which leads to code fragments like this:

What annoys me about this is that I have to define in each handler if it succeeded and what to do if an exception occurred on the database layer. Especially when you have to nest three or more operations this looks ugly. Of course this problem is not new and there exists an alternative approach which is using and composing Java 8 java.util.concurrent.CompletableFuture. By doing so, the same code becomes easier and more readable:

But using CompletableFuture within the Vertx world leads to a problem: Vertx has it’s own threading model to achieve the performance it actually has. On the other hand, some methods of CompletableFuture, e.g. CompletableFuture.supplyAsync(Supplier), run tasks on the common ForkJoinPool which would break the Vertx contract. Open-source-software to the rescue, there is a solution to this problem: VertxCompletableFuture. This special implementation guarantees that async operations run on a Vertx context unless you explicitly specify an Executor in one of the overloaded xyzAsync-methods*.

And here comes even better news: starting from version 2, vertx-jooq also supports this way of dealing with asynchronous database operations by utilizing VertxCompletableFuture. Checkout the new vertx-jooq-future module and the according code generator to create your CompletableFuture-based DAOs.

* There have been discussions in the Vertx developer group about a CompletableFuture based API, e.g. here and especially here. The current status is that they do not provide such API officially, mostly because VertxCompletableFuture breaks the contract of the supplyAsync-methods, since it runs within the Vertx context and not the ForkJoinPool. Also when you pass this CompletableFuture subclass to code that expects a regular CompletableFuture, it breaks the Liskov substitution principle and OOP (thanks for pointing that out in the comments Julien). My opinion is, that if you are using Vertx you are aware of the special threading model and can tolerate that behavior. But, of course, it’s up to you.

Sending push notifications with Vertx and OneSignal

All good things come to an end: that’s what I thought when Parse announced it’s shutdown. Back then, Parse was the best “push notifications as a service”-solution that offered a REST-API to send and schedule cross platform push notifications through a backend service. Because we used it in one of our products, we had to look for a replacement and so I finally stumbled upon OneSignal.

Like Parse, it offers cross platform pushes, a ton of clientside SDKs to easily integrate it in various languages and a solid REST-API to trigger push notifications through a backend. And last but not least, it doesn’t cost any money*.

As you might have noticed in my recent posts I use Vertx a lot. So I thought it would be a good idea to write a library that allows you to send push notifications using OneSignal the Vertx-way. You can see this library as a wrapper around OneSignal’s REST-API that gives you compile-time validation instead of a trial and error approach.

Happy pushing!

* According to the documentation of OneSignal, OneSignal makes money from collecting data of your clients. If you’re concerned, they also offer paid service options.

Vertx loves jOOQ

I’ve recently published a library that connects two frameworks I used a lot in the past: Vertx and jOOQ. Vertx is a “reactive tool-kit for the JVM” that enables you to write non-blocking code and has some nice features built in, like webserver/socketserver-implementation, a message-bus that can run on one or more instances in one network and many more goodies you should check out. On the other hand, if you are coding in Java and you like SQL, there is no way around jOOQ. In my opinion, jOOQ’s two killerfeatures are the possibility to write typesafe SQL and the awesome code-generator that generates POJOs, DAOs and Table-implementations based on your database schema (schema first!). However jOOQ is using JDBC under the hood which blocks the calling thread until the database operation completes.

That is where my library hooks in: it provides a code-generator that adds non-blocking CRUD-methods to all generated DAOs and converter methods that allow you to convert from Vertx’ JsonObject into jOOQ’s POJOs and vice versa.

For code examples, please refer to the github-page.

Vertx + Pebble + Bootstrap = <3

Update


In between, there has been an official implementation of Pebble for Vertx. You should use this dependency in favor of my implementation.


It’s been a long time since I wrote something in here. That is mainly because I moved to a project where no Java was involved and I had to code PHP for quite some time. While I did this, I came closer to web development and the bootstrap-library from twitter. For a new project I decided to switch to Java again, because we had realtime requirements and – to be honest – because I wanted to code in Java again. A lot has happened in between though: Java 8 came out and the whole micro-services train started to move into direction from theoretical discussions to rock-solid technologies which are used in production. One of which is vertx which was released in the third version.

The Goal

In this blogpost I want to show you how you can write a simple chat application using the technologies mentioned above:

  • Vertx: „A polyglot tool-kit to build reactive applications on the JVM”
  • Bootstrap: „The most popular HTML, CSS, and JS framework for developing responsive, mobile first projects on the web“
  • Pebble: „A lightweight but rock solid Java templating engine”

In chat applications there is the requirement that messages have to be pushed from the server to the client. This makes a classic web server stack obsolete since the request-response pattern does not apply. I decided to use WebSockets which are built on top of the well-known HTTP protocol and can be easily implemented on client-side using Javascript.

The WebSocket Server

While I’ve been working on the PHP project, I realized that I don’t have to care about writing web servers. You just write your code and let application servers like apache or nginx do the rest. But we are back in the Java-World. And in the Java-World you have to take care of that. This post is not a deep-dive into every aspect of vertx and I will just explain the parts that are necessary for this post to be understood. If you’re hooked, I recommend you reading their docs. Please note that you don’t have to copy the code snippets one by one, because I’ve setup a github repo for this purpose.

Line 4: First of all, we need a Router. It is used to route the request coming from the vertx-Httpserver. We define a route and handler which is responsible for handling that route so whenever someone is calling „/ws/websocket/„ the handler will be called. The asterisk in this case means that the request could also be „/ws/websocket/asdf“ and would still be handled by that handler.

Line 5: This is how to upgrade a regular request to a WebSockets in vertx (for the sake of completeness: there are also other ways of handling WebSockets in vertx, but I liked this the most).

Line 7: Here we added a handler that is called whenever a message is received on the WebSocket. We expect messages to be binary encoded JSON. For every message we add a senderId-field with a unique identifier for each client to the message and send it to the event bus. The event bus is used to the send messages between several vertx instances – either locally or even clusterwide. But it also works only for one vertx instances like in our case. Every message needs to have an address which is in our case „chat.broadcast“. One thing worth mentioning: there is also a send-method on the eventbus which implies the call of ONE handler, whereas publish will signal all registered handlers.

Line 13: This is how the client will receive messages. We register a consumer on the eventbus that is notified whenever a message is published. We evaluate the notification and check for the senderId-field. If we are not the sender, the message is send to our client.

Line 21: If something unexpected happens, we want to log it.

Line 22: When the client is closed, we have to cleanup everything.

Line 27: Ramp up the HTTP server with the defined route.

That is all logic needed for handling the client-server communication. 

The web server

For convenience reasons, I decided to add the client into the same application. This basically means that we need to have a web server that delivers the HTML-document which has a java-script-client embedded. 

Line 4: We’re working again with a router here. To deliver static content (JavaScript-Libraries, CSS-Content and anything else ’static’) we need a StaticHandler which is part of the Vertx library. We need to define a base-path in which our files are stored. In our case it is “webroot/resources“ which I’ve created under „src/main/resources/„. (The Router is the same that we used for the WebSocket-server. This is important, because if we would create a Router in each Verticle some requests wouldn’t be served. This is because the HttpServer would call the Routers in a round-robin-fashion and therefore not all routes we defined would be known.)

Line 5: Our app will have a fancy favicon, this is how you deliver it.

Line 6: In this example we’re using a templating engine to render our HTML pages. Vertx comes with an out-of-the-box-support of four template engines but I wanted to give pebble a try. The main reason for using Pebble is that I like the templating syntax and it gives a nice performance (https://github.com/mbosecke/template-benchmark) too. Therefore I created my own PebbleTemplate-Engine and added it to the TemplateHandler. The templates are located under „webroot/templates” in this example. 

Line 9: Route every request that has not matched by any other route to our chat_client.html.

Line 10: Ramping up the HTTP server. But wait! Didn’t we use the same port for the WebSocket-server? Yes we did, but vertx is smart enough to combine the routes and handle everything accordingly.

Bring it

We create the server using the good old main-method and instantiate a Vertx-instance as shown. Vertx has the concept of Verticles that provide„a simple, scalable, actor-like deployment and concurrency model“. One benefit is that you can write code as if it was single threaded, because code is always called from the same eventloop. The whole example would totally have worked without Verticles, but creating a Router and everything directly in the main-method. Using Verticles however encourages you to separate your concerns and can be used as an entry-point for developing microservices. Another thing worth mentioning is, that Verticles are deployed asynchronously and therefore the Main-Thread will not block until they are deployed.

The Template

The client is written in JavaScript and embedded into an HTML-document so let’s first look at the document-layout. 

This is the skeleton from which every page will inherit and is basically the start-template provided by bootstrap, however divided into several parts. The include-keyword includes the content of the given source into this document like you would expect (one has to prepend a dot followed by a slash to the source-name in order to make the loading work (at least on my mac)). The HTML itself is pretty basic but I’ll explain the used element classes that come with bootstrap:

  • container-fluid describes a container spanning over the whole document
  • rows are used to define a horizontal group of columns
  • col-md-offset-1 describes that we’ll have 1 column padding on each side without content. The col-md-10 describes a column that has a size of 10/12 of the width of the enclosing row-div since rows are divided in 12 md-columns. If we would add two divs with each having the class col-md-5, we would end up having two columns in one row (plus the two offset-columns). Because bootstrap is mobile first, this is responsive and automatically scales well on all kinds of devices. We could also define the classes not using col-md-x but col-xs-x or col-s-x to have different layouts for devices with extra-small (phones) or small (tablets) devices. For further reading I recommend the bootstrap-docs (http://getbootstrap.com/css/#grid).
  • the last div has the starter-template class which basically moves the content below the navigation header.

Now we are coming to another interesting part of the template engine. We define a block called ‚content‘ which can be overridden by any page that is extending this document. (In our example we only have one site – the chat application – which makes the usage of templates a bit obsolete, but that is another story ;)). We could also add some default values but in this case it is not necessary. The key takeaway from this is that you can write all the boilerplate code in one file and when writing a new pages you just have to add the parts that are new. 

The Client

By using the extends keyword we tell Pebble to inherit from that document. We’re overwriting the content-block with straight-forward HTML. On the bottom of the page, the WebSocket-client logic is defined in Javascript. On Line 45 we’re opening a connection to localhost. Because the server expects and sends binary frames, we set the binaryType accordingly. The rest of the code is not very hard to understand and should be self-explainority. 

The Test

I hosted the complete project on GitHub. The project requires maven and Java 8 to be built properly. After you cloned the project, run Main.java in the IDE of your choice, open two tabs in your browser, point to localhost:8080, select a username and start chatting! As you can see, messages are instantly pushed to the other clients – the benefit of WebSockets. I left some space for modifications though: right now you can only broadcast messages. But what if you want to send a private message to just one recipient? Maybe you want to personalize your account by uploading avatars or securing them with a password on a separate page/template? If I find the time I will cover this in one of my next blog-posts, but until then: happy coding!