Injecting connection information into Typesafe Slick at the last possible point

Often when designing a database-driven application, you will want flexibility as to which database you want to connect to. You will certainly want different databases for production, test, and development environments. You might even want different database engines for different environments. For example, you might use an H2 database in your local development environment, and Postgres in test and production. Or perhaps you are writing an application that will be used in many environments, and you need to support Postgres, MySQL, DB2, Oracle, etc. The usual answer to this problem is dependency injection, and while there are frameworks for this, in many cases, you can accomplish these goals using just the language features of Scala.

Most tutorials (that I have found, anyway) for using Typesafe Slick assume that you are picking one database engine and sticking with it. You need to import implicits specific to the driver you are using. You can easily specify different connection URIs at runtime, but you need to commit to a driver throughout your application.

Here, I will present an alternative that allows you to specify your driver in one place, and have various classes of your database interface code all agnostic to the particular driver you are using. There is no runtime switching of the driver, and (almost) everything is type checked at compile time.

Schema Definition

The Slick docs start with selecting a database driver before doing anything else:

// Use H2Driver to connect to an H2 database
import scala.slick.driver.H2Driver.simple._

But of course, we don’t want to do that. So, here is a schema definition class that only assumes that we will be using some kind of scala.slick.driver.JdbcDriver:

import slick.driver._
class Tables(val profile: JdbcProfile) {
  import profile.simple._
  def ddl: profile.SchemaDescription = names.ddl

  class PersonTable(tag: Tag) extends Table[(Long, String)](tag, "person") {
    def id = column[Long]("id", O.PrimaryKey, O.AutoInc)
    def name = column[String]("name")
    def * = (id, name)
  val names = TableQuery[PersonTable]

Note that this gets the implicits from the JdbcProfile that is passed in. This allows us to pass in the profile when we know what it is, but we don’t have to worry about it here.

DAO layer

Next, lets add some DAO code that uses this new table:

class PersonDao(t: Tables) {
  import t.profile.simple._
  def makePerson(name: String)(implicit sess: Session): Long =
    (t.names returning += (-1, name)

Again, we just import the implicits needed for the Slick DSL from the profile that was given to the schema definition.

Finally, we bind the database driver

Up until now, we have been agnostic as to which database driver we use. At some point, though, you need to choose one. Here is a simple Main class that does that with a hardcoded value. You can imagine some logic that reads a config file, or some other mechanism to do this, though.

import slick.driver._

object Main extends App {
  import scala.slick.driver.H2Driver.simple._
  import slick.driver.H2Driver.profile
  val db = Database.forURL("jdbc:h2:db", driver="org.h2.Driver")
  db.withSession { implicit s => 
    val foo = new PersonDao(new Tables(profile))

Only lines 4-6 in the above snippet commit us to a specific database. They could be swapped out for a different driver, with a different profile, and a different connection string, and our schema definition and DAO code could remain unchanged.

While coming up with this, I did find one case where the compiler got confused, and I needed to cast something, though. This was while adding a function to recreate the DDL:

  db.withSession { implicit s => 
    val ddl: profile.SchemaDescription = new Tables(profile).ddl.asInstanceOf[profile.SchemaDescription]
    import profile.Implicit._

From looking at the code, you can tell that the Tables.ddl object should be of the type profile.SchemaDescription (for whatever type profile you are passing in), but the cast was necessary to make the compiler happy.

Caveats to this approach

  • You are stuck with the lowest common denominator. Some features may compile, but then fail at runtime if you use a feature that one driver doesn’t support. Likewise, if one driver presents a non-standard feature, you may not be able to use it.
  • Be sure to run your test suite (at least the part that exercises your db interface) against all drivers you intend to support. If your database functions prove to behave identically whether run on an H2 in-memory database or a real Postgres DB, you can then test the rest of your code (that just calls your db interface layer) against the H2 in-memory db, and possibly achieve faster test runs.
Posted in computers | Tagged , | Leave a comment

33 is the Magic Number…

Photo courtesy of Flickr user schoeband

On the No Agenda Show, there is the theory that the number ’33′ is inserted into news stories as code.  No one is sure what the code means, though.  In some contexts, it seems to mean that the rest of the news story is bogative, or maybe it means ‘mission accomplished’, or it is just a flag to alert people in-the-know to take heed.

In any case, I created a little site to track instances of such numbers in the news.  Introducing:

Subscribe to the feeds if you want to be kept abreast of any possible coded stories!

This was a fun little project to get more familiar with Heroku, Play 2.1, and Reactive Mongo.  I even worked a little bit of Shapeless HLists in there, for good measure.

Posted in Uncategorized | Leave a comment

FIBS: Functional interface for Interactive Brokers for Scala

Announcing a small side project I have been hacking on: a Scala wrapper library for the Interactive Brokers TWS API.  The TWS API uses a message-passing design, where you pass a message asking for a quote, for instance, and then you get a series of messages back, making up that quote.  It is up to you to keep track of which response messages are referring which quote request, and the whole thing involves a lot of mutable state.

I, however, wanted to write an application staying as functionally pure as possible, with little or no shared mutable state, which could process many quotes at once.  I wanted to do this using my favorite functional language du jour, Scala/scalaz.

So, I created FIBS, which is a terrible backronym that I am claiming stands for Functional interface for Interactive Brokers for Scala.  It is in its infancy, and currently only supports the following operations:

  • Realtime Stock Quotes
  • Historical Stock Quotes

There is much, much more in the API to flesh out.  One of the other things I don’t like in the TWS API is that it is up to you to know which parameters are appropriate for the request you are making.  (For example, only certain order parameters make sense for a given order type.)  So, I want to enforce this logic with the type system.  You, as the consumer fo my API, should never pass a null value because that parameter makes no sense in the context of what you are doing.  The type system should enforce this.

Instead of waiting for various pieces of information to return, you will get a Promise or a Stream, or some other monad to allow you to move forward with your code while IB is doing its thing.

So, I am making progress on the API slowly, as I need it for my application.  I wanted to make it public to see if anyone else was interested.  I could also use some domain expertise in knowing exactly what parameters are appropriate, when.  My experience as a quant is strictly amateur.

Check it out on Bitbucket!

Posted in computers | Tagged , , , , | Leave a comment

Building Inline Comments for Pull Requests and Commits

(or, what I did over summer vacation)

For the last few months, I have been working with the Bitbucket team at Atlassian.  I switched over to this team at the beginning of the summer to help build a new inline commenting feature on pull requests and commit pages, to help make the tool more useful for code reviews in a team.

It was a great project, and I wrote up a story of how we built inline comments for pull requests and commits over at the Bitbucket blog.  Check it out!

Posted in Uncategorized | Leave a comment

Proof that Netflix is the only thing holding the USPS afloat

"Netflix Only" chute at the post office

Posted in Uncategorized | Leave a comment

LivingSocial/Fiesta Americana Resort Bait & Switch Scam

UPDATE: I contacted LivingSocial about this issue, and they responded in a timely manner, and offered a partial refund, which I feels remedies the situation.  I am leaving this post up to tell the story, and to alert people that they should always be on guard for issues like this, but at least LivingSocial has made a good effort to rectify the situation.
My wife and I recently returned from a trip to Los Cabos, Mexico, where we stayed at the Fiesta Americana Grand Los Cabos Golf & Spa Resort.  We purchased the resort package through “Living Social Escapes” (archived as PDF in case Living Social removes the page).  As excellent of a time we had while enjoying the warm weather and the beautiful beach, some of the items included in the offer we felt were misleading and blatantly untrue.

Under one of the bullets in the “escape kit” it states: “Daily Continental Breakfast, Afternoon Cocktails, and Hors D’œuvres at the Grand Club and Free Upgrade to Grand Club Level for First 50 Vouchers Sold”.  This gives the impression that daily continental breakfast, afternoon cocktails and hors D’oeuvres are included in the package deal and that the first 50 vouchers sold will receive free upgrade to Grand Club Level.  After being denied the continental breakfast, we spoke to the front desk.  The front desk people read it as “all of the items in this statement are only for the first 50 people.  This is a very tricky and misleading statement. Furthermore, in the body of the webpage, it states

“You’ll know you’ve been good this year as you enjoy meet-and-greet services and private ground transportation from Los Cabos International Airport, private check-in and check-out service, daily continental breakfast, and a $600 resort experience credit.”

This states nothing about the “first 50 vochers sold”.  We probably would not have bought the package had we know that the continental breakfast was not included, and instead found a package that included the amenities we were looking for at the right price.

Under another bullet it states:  “In-Room Bottle of Wine and Canapés Daily”.  When we first arrived, there was a bottle of wine and a plate of canapés in our room waiting for us.  The reminder of the days, there was no wine or canapés.  When we asked the front desk about this, they told us that a bottle of wine was only supplied on the first day and that Living Social lied about that in their ad. One front desk woman even admitted that the statement was a bait and switch tactic.

Overall, we are very unimpressed with Living Social’s misleading tactics.  We would love to be able to take advantage of the wonderful offers they have available, however they have lost a lot of our trust.

Posted in Uncategorized | Tagged | Leave a comment

Accessing Erased Type Parameter Information in Scala

One of the things holding scala back from being a more robust language is the fact that it runs on the JVM. (On the other hand, this is also one of its strengths–you can easily interoperate with existing Java code and libraries, as well as any other code that runs ont he JVM, like Groovy or Clojure.) Because Scala runs ont he JVM, it suffers from type erasure, which means that any generic type parameters that are in the code are lost after compilation, and are no longer present in the byte code, and thus at runtime. (This was a design decision made by the Java team when they introduced generics in Java 1.5, in order to preserve byte code backward-compatibility.)

In order to work around this, Scala introduced the Manifest class, which captures the type information during compilation, and allows you to access this type at runtime.  This was originally designed to allow for the creation of arrays of the generic type at runtime, but it can also be used for other custom generic types.  It uses implicit parameters to capture this information.

The problem (that I had), though is that you can’t seem to access a class’s type parameters directly in a method of that class.  So, this does’t work:

import scala.reflect.Manifest
class MyClass[T: Manifest] {
  def myType(implicit m: scala.reflect.Manifest[T]) = m.toString

You won’t get a compilation error, but you will just get ‘Nothing’ as your type inside your method:

scala> new MyClass[Long]
res17: MyClass[Long] = MyClass@2769aba2
scala> res15.myType
res18: java.lang.String = Nothing

This is because there was no type parameter to the method itself, and the [T] in the method’s implicit argument is not the same [T] that is used to parameterize the class.  If we add a type parameter to the ‘myType’ method, we can get closer to what we want:

class MyClass2[T: Manifest] {
  def myType[U](implicit m: scala.reflect.Manifest[U]) = m.toString

Then, we can try it out:

scala> new MyClass2[Long]
res19: MyClass2[Long] = MyClass2@6c6455ae
scala> res19.myType[Int]
res20: java.lang.String = Int

Okay, so now we are getting some type information out, but it really isn’t ideal.  Note that the type parameter inside the ‘myType’ method was ‘Int’, which was the parameter we passed to the method, not ‘Long’, which is what we passed to the class.  What if we want to get the class’s type parameter?  We certainly don’t want the calling code to have to keep track of the type parameters in multiple places (once at the constructor call, and once more at the method invocation).  So, we can add a second method to help smooth this out:

class MyClass3[T: Manifest] {
  def myType[U](implicit m: scala.reflect.Manifest[U]) = m.toString
  def myTypeWtihoutExtraParam = myType[T]

And, then, we get:

scala> new MyClass3[Long]
res30: MyClass3[Long] = MyClass3@543d8ee8
scala> res30.myTypeWtihoutExtraParam
res31: java.lang.String = Long

So, you can see that the type that was passed in to the class constructor as a generic parameter is now available in the method, without the method caller needing to supply it after the object is created.

A couple of things to note:

  • Don’t forget to import scala.reflect.Manifest
  • The type parameter in the constructor definition is context-scoped, so the ‘: Manifest’ part is important.  This technique won’t work if you just specify the parameter as ‘[T]‘.  If you leave this off, you will get an error that reads, “No Manifest available for T.”
Posted in computers | Tagged , , , | 1 Comment