CoronaMQ

It’s been a while since I started this pet-project of mine: it was a proof-of-concept message queue that uses PostgreSQL, the vertx-postgres-driver and it’s NOTIFY LISTEN support. The idea was, that if a row was added in to a table (jobs, tasks, you name it), Postgres would send a notification to the driver. The driver would then send this notification, alongside with the task’s payload, over the vertx eventbus and some worker would react to it. The following figure visualizes this concept:

This was already two years ago and since then I didn’t invest more time into it – like with any other sideproject you start. But then, two things happened: vertx added support for Future-APIs in their code-generator and I caught Covid – no joke. I started to play around with the code-generator and created APIs for mutiny and rxjava. I read about and added vertx-service-proxies for the TaskRepository, decoupled the participants (Broker, TaskRepository and Workers) using vertx-service-discovery, added metric-profiling with Mircometer and started using Testcontainers for the integration tests.

Now, again, a couple of months later, I finally found some time to upload the project to maven-central and share it with the world. I’m happy with the result and think it’s a cool real-world-application that uses awesome open source libraries. If you have any feedback, I’m happy to hear it, either in the comments, on Twitter or on the github-page.

Happy coding!

vertx-jooq 5.1.0 now on maven central

This release comes with nice benefits for the reactive modules. Users had a hard time in the past if they wanted to map a JSONB-column to a custom POJO. If they created a custom jOOQ-Converter, the generated entity would map the field to the POJO as expected, but they couldn’t really work with it, because the mapping between the POJO and the reactive-driver data types was unknown. For that matter, there is now a new PgConverter that you can use in the reactive modules. Checkout this example for a how-to.

More information about the release can be found at the project’s github page.

vertx-push-onesignal 1.7 released

When working with OneSignal, you sometimes receive an API response telling you that the notification you’ve sent does not have any recipients (“All players are not subscribed”). This can either mean that the notifications you’ve setup are not working like intended (e.g. you’re using wrong filters) or that a former active user of yours became inactive and cannot be addressed any longer. In case you are targeting users using tags this can happen quite frequently.

To avoid the cost of creating a OneSignalException in that case there is an option now to prevent vertx-push-onesignal from creating these exceptions in case that response is received from the OneSignal API.

See my github-page for an example how to set it up properly.

vertx-jooq-async 0.4 released!

vertx-jooq-async the (un)beloved child of vertx-jooq has been released in version 0.4! This version finally implements insertReturningPrimaryAsync in all VertxDAOs*.

In addition, the complex code in the DAOs default methods has been abstracted and moved to vertx-jooq-async-shared. That means less code duplication and fewer points of failure – a move that will very likely be done in vertx-jooq soon as well.

Finally the vertx-dependency version has been bumped to 3.5. (and so the support for RXJava2).

* Unfortunately, this only works for MySQL and numeric auto-increment keys. That is, because the implementation is based on the implementations of io.vertx.ext.asyncsql.impl.AsyncSQLConnectionImpl and only the MySQL variant returns the generated ID.

vertx-jooq 2.4 released

Note: Unfortunately, the dependencies to rxjava1 haven’t been removed completely in 2.4.0. RX users please checkout version 2.4.1.

I’m happy to announce the release of vertx-jooq 2.4. The main driver behind this release was the dependency upgrade to vertx 3.5 and thus the change of the rx-dependency in vertx-jooq-rx from rxjava to rxjava2. In addition, the JSON key names of the generated POJO are now the same as their database counterparts. This change has been announced here. Last but not least, a bug has been fixed when dealing with JsonObjectConverter or JsonArrayConverter.

Converting a jOOQ-Record into a vertx JsonObject

In vertx-jooq, there are a couple of ways to convert a jOOQ Record into a vertx JsonObject. First, checkout the straightforward way, which involves two steps: load the POJO using a generated DAO and then convert it with the toJson method.

	public class SomeJsonTest {
	public void getJsonFromPojo(){
	SomethingDao somethingDao = new SomethingDao();
	//setConfiguration/ setVertx left out
	CompletableFuture<JsonObject> jsonFuture = somethingDao
	.fetchOneBySomeidAsync(1)
	.thenApply(ISomething::toJson);
	}
	}

view raw

SomeJsonTest.java

hosted with ❤ by GitHub

It turns out however that on the way to a JsonObject, a lot of copying is going on. First the SELECT statement is mapped to a Record, then converted to a POJO and finally converted into a JsonObject. The code below is copied from DAOImpl class which is extended by every VertxDao:

	public abstract class DAOImpl<R extends UpdatableRecord<R>, P, T> implements DAO<R, P, T> {
	// …
	@Override
	public /* non-final */ <Z> P fetchOne(Field<Z> field, Z value) {
	R record = using(configuration)
	.selectFrom(table)
	.where(field.equal(value))
	.fetchOne();
	return record == null ? null : mapper().map(record);
	}
	// …
	}

view raw

DAOImpl.java

hosted with ❤ by GitHub

If you’ve enabled the generation of interfaces in vertx-jooq (which is the default), then you could convert the Record directly into a JsonObject without converting to POJO first.

	public class SomeJsonTest {
	public void getJsonFromRecord(){
	SomethingDao somethingDao = new SomethingDao();
	//setConfiguration/ setVertx left out
	CompletableFuture<JsonObject> jsonFuture = somethingDao.executeAsync(dslContext -> dslContext
	.selectFrom(somethingDao.getTable())
	.where(Tables.SOMETHING.SOMEID.eq(1))
	.fetchOne())
	.thenApply(ISomething::toJson)
	;
	}
	}

view raw

SomeJsonTest.java

hosted with ❤ by GitHub

The major disadvantage of this solution is that you have to write the SELECT statement by yourself. On the other hand, you save the creation of the POJO, which is a plus. But what if we’re joining on another table so we cannot directly map into a generated Record? See this example:

	public class SomeJsonTest {
	public void getJsonFromRecord2(){
	SomethingDao somethingDao = new SomethingDao();
	CompletableFuture<JsonObject> jsonFuture = somethingDao.executeAsync(dslContext -> dslContext
	.select(Tables.SOMETHING.SOMEID)
	.select(Tables.SOMEOTHERTHING.SOMEVALUE)
	.join(Tables.SOMEOTHERTHING)
	.on(Tables.SOMETHING.SOMEID.eq(Tables.SOMEOTHERTHING.SOMEOTHERID))
	.where(Tables.SOMETHING.SOMEID.eq(1))
	.fetchOneMap())
	//or .fetchOne(Record::intoMap)
	.thenApply(JsonObject::new);
	}
	}

view raw

SomeJsonTest.java

hosted with ❤ by GitHub

Because the fetched Record is dynamic, we cannot simply call toJson on it. Instead we call the fetchOneMap method which “returns at most one resulting record as a name/value map.” Luckily, JsonObject has a constructor taking a Map<String,Object> which is exactly the same generic type returned (it even returns the same Map implementation that JsonObject uses under the hood: LinkedHashMap). It is noteworthy that this could even be done with pure jOOQ and vertx (leaving the execution aside). Some caveats though:

1. No conversion happens. If you use a custom generator to generate the schema objects and have overwritten handleCustomTypeToJson / handleCustomTypeFromJson, you’re probably in bad shape. If one of the fields you’re fetching is one of those types, this conversion is bypassed.
2. Because the constructor makes no checks and takes the map ‘as is’ without copying, you’re probably adding illegal JSON-types.
3. The Map‘s key will have the name of the database column it is representing. This differs from the default-setting in vertx-jooq, which uses the name of the POJO’s member variable representing that column and thus would produce different JSON compared to the toJson method for the same Record/POJO. Starting with 2.3.5 however this can be changed so the keys are rendered with the same syntax.

Lessons learned

When simple CRUD is not enough, you have to fetch some joined Records and need to convert the results into a JsonObject, you should change the syntax how the JsonObject's keys are rendered in the POJO (3.). This is the first step in making both ways of generating JSON interoperable.

Things get trickier if you have to deal with conversion of custom JSON types. If your app/service is mostly reading data, you could handle conversion already on jOOQ-level so your converted type is already a valid JsonObject type. For example, if you’re using a SQL DateTime field “lastUpdated” on database level, just write a converter that formats the DateTime as a String. In turn, both the generated POJO and the fetchOneMap/fetchMaps methods will return a String for the “lastUpdated” entry and produce same results.

This can become a problem when those converted values need to be altered by the same app: a) it is more convenient and less error-prone to set a java.time.LocalDateTime object instead of dealing with Strings and b) some of the types may have special SQL-functions (e.g. dateAdd) which you cannot use any longer for that type.

Conclusion

Surprisingly there is no golden rule. First, I recommend to change the syntax of how JSON keys are rendered to be the same as the database column names (I will most likely change that behavior to the default in one of the next releases). This means you can use the fetchOneMap/fetchMaps methods most of the time to produce JSON. When dealing with custom types you should check how frequently those are updated by your app. If you’re facing a read-only app, write custom jOOQ converters for these types. If these types are altered quite often, you should stick to their actual type and handle the conversion into a proper JSON value on your own.

Hello async my old friend

Let me present you vertx-jooq-async: the world’s first fully async and type-safe SQL code generation and execution tool for VertX™. Async? Wasn’t it asynchronous before? It was, but what the code did was wrapping JDBC-calls using Vertx’ executeBlocking-method, which just shifts the blocking code from one event loop to another (see also).

With this release however, things have changed. jOOQ is still used for generating code and creating type-safe queries, but code is executed utilizing vertx-mysql-postgresql-client. That library in turn is based on a non-blocking driver for MySQL and Postgres databases, so this time it is really non-blocking.

Although some stuff is yet missing – you should go and check out the github-page. Now.

The Future Is Here

Getting asynchronous operations right is hard. And when you have to pass a result of one function to the next, it can only get worse. Since today, vertx-jooq’s API for instance only allowed asynchronous operations with a callback handler, which leads to code fragments like this:

	//fetch something from A to make a lookup on B
	ADao adao = new ADao();
	adao.findByIdAsync(123,h1->{
	if(h1.succeeded()){
	BDao bdao = new BDao();
	bdao.findBySomeAValueAsync(h1.result().getSomeAValue(),h2->{
	if(h2.succeeded()){
	//do something with the result
	}else{
	//oops something went wrong
	}
	});
	}else{
	//oops something went wrong
	}
	});

view raw

AsyncHandlerExample.java

hosted with ❤ by GitHub

What annoys me about this is that I have to define in each handler if it succeeded and what to do if an exception occurred on the database layer. Especially when you have to nest three or more operations this looks ugly. Of course this problem is not new and there exists an alternative approach which is using and composing Java 8 java.util.concurrent.CompletableFuture. By doing so, the same code becomes easier and more readable:

	//fetch something from A to make a lookup on B
	ADao adao = new ADao();
	CompletableFuture<A> aFutureLoaded = adao.findByIdAsync(123);
	BDao bdao = new BDao();
	aFutureLoaded.thenCompose(
	a->bdao.findBySomeAValueAsync(a.getSomeAValue());
	).whenComplete((bResult,ex)->{
	if(ex==null){
	//do something with the result
	}else{
	//oops something went wrong
	}
	});

view raw

CompletableFutureExample.java

hosted with ❤ by GitHub

But using CompletableFuture within the Vertx world leads to a problem: Vertx has it’s own threading model to achieve the performance it actually has. On the other hand, some methods of CompletableFuture, e.g. CompletableFuture.supplyAsync(Supplier), run tasks on the common ForkJoinPool which would break the Vertx contract. Open-source-software to the rescue, there is a solution to this problem: VertxCompletableFuture. This special implementation guarantees that async operations run on a Vertx context unless you explicitly specify an Executor in one of the overloaded xyzAsync-methods*.

And here comes even better news: starting from version 2, vertx-jooq also supports this way of dealing with asynchronous database operations by utilizing VertxCompletableFuture. Checkout the new vertx-jooq-future module and the according code generator to create your CompletableFuture-based DAOs.

* There have been discussions in the Vertx developer group about a CompletableFuture based API, e.g. here and especially here. The current status is that they do not provide such API officially, mostly because VertxCompletableFuture breaks the contract of the supplyAsync-methods, since it runs within the Vertx context and not the ForkJoinPool. Also when you pass this CompletableFuture subclass to code that expects a regular CompletableFuture, it breaks the Liskov substitution principle and OOP (thanks for pointing that out in the comments Julien). My opinion is, that if you are using Vertx you are aware of the special threading model and can tolerate that behavior. But, of course, it’s up to you.

Sending push notifications with Vertx and OneSignal

All good things come to an end: that’s what I thought when Parse announced it’s shutdown. Back then, Parse was the best “push notifications as a service”-solution that offered a REST-API to send and schedule cross platform push notifications through a backend service. Because we used it in one of our products, we had to look for a replacement and so I finally stumbled upon OneSignal.

Like Parse, it offers cross platform pushes, a ton of clientside SDKs to easily integrate it in various languages and a solid REST-API to trigger push notifications through a backend. And last but not least, it doesn’t cost any money*.

As you might have noticed in my recent posts I use Vertx a lot. So I thought it would be a good idea to write a library that allows you to send push notifications using OneSignal the Vertx-way. You can see this library as a wrapper around OneSignal’s REST-API that gives you compile-time validation instead of a trial and error approach.

Happy pushing!

* According to the documentation of OneSignal, OneSignal makes money from collecting data of your clients. If you’re concerned, they also offer paid service options.

Vertx loves jOOQ

I’ve recently published a library that connects two frameworks I used a lot in the past: Vertx and jOOQ. Vertx is a “reactive tool-kit for the JVM” that enables you to write non-blocking code and has some nice features built in, like webserver/socketserver-implementation, a message-bus that can run on one or more instances in one network and many more goodies you should check out. On the other hand, if you are coding in Java and you like SQL, there is no way around jOOQ. In my opinion, jOOQ’s two killerfeatures are the possibility to write typesafe SQL and the awesome code-generator that generates POJOs, DAOs and Table-implementations based on your database schema (schema first!). However jOOQ is using JDBC under the hood which blocks the calling thread until the database operation completes.

That is where my library hooks in: it provides a code-generator that adds non-blocking CRUD-methods to all generated DAOs and converter methods that allow you to convert from Vertx’ JsonObject into jOOQ’s POJOs and vice versa.

For code examples, please refer to the github-page.