4/28/09

My take on 99 problems in Scala (first 22)

A while ago I saw 99 problems in Prolog they looked really good and I started trying to solve them in Scala. Now, thanks to Tony Morris I found that there's a Scala version!
Nice stuff to get your skills up to shape. It helped me to improve my understanding of folds, maps, flatMaps, zips, etc. Also I learned new pattern matching techniques (tuples and guards).
You must really give the exercises a shot!
Here are my solutions:



package exercises

object NinetyNine {
  //P01
  def last[A](xs: List[A]): A = xs match {
    case x :: Nil => x
    case x :: xs => last(xs)
    case Nil => throw new NoSuchElementException
  }

  //P02
  def penultimate[A](xs: List[A]): A = xs match {
    case x :: y :: Nil => x
    case x :: xs => penultimate(xs)
    case _ => throw new NoSuchElementException
  }

  //P03
  def nth[A](i: Int, xs: List[A]): A = xs match {
    case x :: xs => if (i == 0) x else nth(i - 1, xs)
    case _ => throw new NoSuchElementException
  }

  def nth2[A](i: Int, xs: List[A]): A = xs match {
    case x :: xs if (i == 0) => x
    case x :: xs if (i > 0) => nth(i - 1, xs)
    case _ => throw new NoSuchElementException
  }

  //P04
  def length[A](xs: List[A]): Int = xs match {
    case x :: xs => 1 + length(xs)
    case Nil => 0
  }

  //Although straight recursion is fun, using folds is more effective
  // folds are abstractions for recursion over data structures

  //P04
  def length1[A](xs: List[A]): Int = xs.foldLeft(0)((a, b) => a + 1)

  //P05
  def reverse[A](xs: List[A]): List[A] = xs.foldLeft(Nil: List[A])((a, b) => b :: a)

  //P06
  def isPalindrome[A](xs: List[A]): Boolean = reverse(xs) == xs

  //P07
  def flatten[A](xs: List[List[A]]): List[A] = xs.foldRight(Nil: List[A])(
    (xs, acc) => xs match {
      case ys: List[A] => ys ::: acc
      case y: A => y :: acc
    })

  //P08
  def compress[A](xs: List[A]): List[A] =
    xs.foldRight(List[A]())((y, ys) =>
      if (ys != Nil && ys.head == y) ys else y :: ys)

  //P09
  def pack[A](xs: List[A]): List[List[A]] = xs match {
    case Nil => Nil
    case _ => xs.takeWhile(_ == xs.head) :: pack(xs.dropWhile(_ == xs.head)) //can use span too, see P13
  }

  //P10
  def encode[A](xs: List[A]): List[(Int, A)] =
    pack(xs).map(y => (y.length, y.head))

  //P11
  def encodeModified[A](xs: List[A]): List[Any] =
    pack(xs).map(
      y => if (y.length == 1) y.head else (y.length, y.head))

  //Type safe version
  def encodeModifiedTypSafe[A](xs: List[A]): List[Either[A, (Int, A)]] =
    pack(xs).map(
      y =>
        if (y.length == 1) Left(y.head)
        else Right(y.length, y.head))

  //P12
  def decode[A](xs: List[(Int, A)]): List[A] = {
    def expand(i: Int, a: A): List[A] = if (i == 0) Nil else a :: expand(i - 1, a)
    flatten(xs.foldRight(List[List[A]]())((y, ys) => expand(y._1, y._2) :: ys))
  }
  //is better with flatMap
  def decode1[A](xs: List[(Int, A)]): List[A] =
    xs.flatMap(ny => List.make(ny._1, ny._2))

  //P13
  def encodeDirect[A](xs: List[A]): List[(Int, A)] = xs match {
    case Nil => Nil
    case _ => {
      val ys = xs.span(_ == xs.head)
      (ys._1.length, xs.head) :: encodeDirect(ys._2)
    }
  }

  //P14
  def duplicate[A](xs: List[A]): List[A] =
    xs.foldRight(Nil: List[A])((y, ys) => y :: y :: ys)
  //is better with flatMap
  def duplicate1[A](xs: List[A]): List[A] = xs.flatMap(y => y :: y :: Nil)

  //P15
  def duplicateN[A](n: Int, xs: List[A]): List[A] =
    xs.foldRight(Nil: List[A])((y, ys) => (1 to n).toList.map(_ => y) ::: ys)
  //is better with flatMap
  def duplicateN1[A](n: Int, xs: List[A]): List[A] =
    xs.flatMap(y => List.make(n, y))

  //P16
  def drop[A](n: Int, xs: List[A]): List[A] =
    xs.zipWithIndex.filter(_._2 % n != 0).map(_._1)

  //P17
  def split[A](n: Int, xs: List[A]): (List[A], List[A]) =
    xs.splitAt(n) // or (xs.take(n),xs.drop(n))

  //ok, that was too easy, let's try recursion
  def splitRec[A](n: Int, xs: List[A]): (List[A], List[A]) =
    (n, xs) match {
      case (0, _) => (Nil, xs)
      case (_, y :: ys) => {
        val rec = splitRec(n - 1, ys)
        (y :: rec._1, rec._2)
      }
      case (_, Nil) => (xs, Nil)
    }

  //P18
  def slice[A](s: Int, e: Int, xs: List[A]): List[A] =
    xs.slice(s, e) // or xs.drop(s).take(e-s)

  //with recursion
  def sliceRec[A](s: Int, e: Int, xs: List[A]): List[A] =
    (s, e, xs) match {
      case (0, 0, xs) => Nil
      case (0, _, y :: ys) => y :: sliceRec(0, e - 1, ys)
      case (_, _, y :: ys) => sliceRec(s - 1, e - 1, ys)
    }

  //P19
  def rotate[A](n: Int, xs: List[A]): List[A] = {
    val s = split((if (n > 0) n else n + xs.length), xs)
    s._2 ::: s._1
  }

  //P20
  def removeAt[A](n: Int, xs: List[A]): (List[A], A) = {
    val (heads, tails) = split(n, xs)
    (heads ::: tails.tail, tails.head)
  }

  //P21
  def insertAt[A](e: A, n: Int, xs: List[A]): List[A] = {
    val (heads, tails) = split(n, xs)
    heads ::: e :: tails
  }
  //with recursion
  def insertRecAt[A](e: A, n: Int, xs: List[A]): List[A] =
    (n, xs) match {
      case (0, ys) => e :: ys
      case (_, y :: ys) => y :: insertRecAt(e, n - 1, ys)
    }

  //P22
  def range[A](s: Int, e: Int): List[Int] = (s to e).toList

  //I don't think is the purpose of the exercise! Again, let's try recursion
  def rangeRec(s: Int, e: Int): List[Int] = if (e - s == 0) e :: Nil else s :: rangeRec(s + 1, e)

  //recursion and pattern matching with guards
  //a little more readable
  def rangeRecPm(s: Int, e: Int): List[Int] = (e - s) match {
    case x if (x > 0) => s :: rangeRecPm(s + 1, e)
    case x if (x == 0) => e :: Nil
    case _ => error("Invalid range")
  }
}
[EDIT: fixed problem 20 ]

4/7/09

Spare change at the whole Static-vs-Dynamic thingy


(NOTE: I'm a software developer, I have a blog, clearly I MUST write about static-vs-dynamic ... I guess soon I'll be writing a monads tutorial)

With all the fuss about Twitter migrating (the back-end) from Ruby to Scala (w00t!), the whole static-vs-dynamic debate has been re-ignited.
As a disclaimer, I'm in the static typing "camp". I've tried Ruby but didn't spark anything. In "dynamic languages", I have more experience with JavaScript (and for the record, I think is awesome: a true underdog of programming languages).
While I can appreciate the freedom and speed of development of dynamic languages, and I think is a matter of personal styles than about which one is better ("choose your own poison"), I think static typing suits better my frame of mind while creating software.
Anyhow, I felt something was missing in the debate and Twitter's use of "kind_of?" points to one of the reasons for static typing I been trying to articulate for a while.
(First and foremost: if you're going to complain "static typing forces me to add a lot of boilerplate and cruft", look into Haskell, ML, Scala, or F# and then come back or you'll force me to do evil things to cute creatures. And if you want to learn how a type system can be used to PROVE that a program is correct, look into Coq)

Types reduces the possible combinations between components
Any software program can be seen as a system of interconnected elements (modules/functions/objects, etc.), by adding type information you're restricting the ways those elements can connect to each other and the possible execution paths of the system. By doing that, you're loosing some of the valid combinations, you're also trimming many invalid ones.
The argument is with good unit tests you can check for the same things, but you have to actually write the code for that. While you should have unit tests for your module, without type restrictions, you have to write and test more execution paths. Moreover, writing the test, executing it and analyzing the result are separate steps, where your brain has to switch contexts.

Types as a Poka-Yoke
There's a lot of influence of Lean Manufacturing in software development (all the "Agile" stuff ), and from that perspective, having more restrictions on how you can use the available components acts like a poka-yoke mechanism: "a behavior-shaping constraint, or a method of preventing errors by putting limits on how an operation can be performed in order to force the correct completion of the operation". The same Lean / Total Quality Management recommends catching defects as early as possible and not use testing as a product quality measure. How more agile can you be when you don't have to write and run test to achieve a basic level of integration quality?

Types as (kind-of) contracts
By using static typing you get self-documenting self-enforcing operation contracts (for a price)
. I believe the larger and more complex the system, the more static typing can help than hinder your work.
NASA could have saved some money if they were able to write:

class MarsObiter {
def setThrust(v:Thrust[MetricUnits])= ...

There's no way you can use a Thrust[ImperialUnits] instead. (For extra points, you can define an implicit conversion in Scala :D )
But even if you have a basic type system like Java, don't just suffer it, use it at your advantage, create the types that can help you to write better code.
Check Stephan's excellent post on what do you loose if you use only primitive types with no semantic meaning (or no types at all)

Enough!
But enough babbling, I can show some examples using the "lowly" Java language:
Suppose we have a method to book a shipment for a customer and a container on a specific date.
Let's start without any typing information:


public bookShipment(customerId, containerId, shipmentDate){...


Totally valid, but... is "1234-321" a valid customerId? can I use a string as shipmentDate? in which format? What I get back?
Let's improve things a little by adding some basic typing information:


public boolean bookShipment(int customerId, int containerId, Date shipmentDate){...


Well, now I now that "X123Z" is not a valid customer or that "MRSK0023" is not a valid container, the date must be a proper date object, and it doesn't take a PhD to infer that will return true if the booking is successful.
But still, nothing prevents me to swap customerId with containerId and get a nasty but subtle bug. Wouldn't be nice if the compiler can prevent that? Well, it can if you don't use primitives and use more domain meaningful types, giving you a DDD flavor (and that's a win-win situation)


public boolean bookShipment(Customer customer, Container container, Date time){...


By examining the signature we know we can't pass a Container when a Customer is expected, and the method reflects better the domain of the application and tries to move towards "an ubiquitous language". But we can improve that:


public bookShipment(customer, container, shipmentDate) { ...


Here Hindley-Milner type inference takes care of identifying the input and output paremeters. No need to declare a thing and still type safe!
(although for documentation/readability purposes, I prefer explicit type declarations, and seems that the Haskell community agrees... that's why I don't mind typing the types of methods in Scala)



The whole point about all this? Go back and get things done, use whatever suites you best and don't waste your time reading blogs and stuff ;-)
If you excuse me, I need to step down from my tittle soapbox and go to do some coding...

4/1/09

Quote of the month

"Bad programming is easy. Idiots can learn it in 21 days, even if they are dummies."

2/8/09

Flavors of Curry: Scala vs Haskell


Scala is a nice language to explore functional programming, and for me it was easy to use it as a “stepping stone” and jump to Haskell with all the purity and laziness (I think is because I'm the kind of guy that enter slowly in the cold water...). I'm using Haskell to learn the theory and tools of functional programming (in a language that forces you to), so I can use them in Scala.

To that end, I'm going through “Learn You a Haskell for Great Good!” (I seem to have a fondness for technical books with silly images). Usually I'll try to do the same exercises in Scala to compare.

I was going through the chapter about higher order functions and currying. What are curried functions? You can find a very good explanation in wikipedia (of course!). Basically is the transformation of a function with multiple parameters into another function with only one parameter that returns a function of the remaining parameters. With that, your language can care only about functions with only one parameter and still be equally powerful. The advantage is if you call a function with less parameters, you get a partially applied function.

One of the examples on the “Learn you...” is a “flip” function: a function that exchanges the first and second parameter of a given function. In Haskell is as easy as:

flip1 f x y = f y x

That's all! Hindley-Milner type inference ensures the type safety:

flip1 :: (t1 -> t -> t2) -> t -> t1 -> t2

As an example:


f x y =x/y
f 1 2
0.5

g= flip1 f
g 1 2
2.0


BTW, there's something odd:


Prelude> let flip1 f x y = f y x
Prelude> :t flip1
flip1 :: (t1 -> t -> t2) -> t -> t1 -> t2
Prelude> let f x y =x/y
Prelude> :t f
f :: (Fractional a) => a -> a -> a
Prelude> let g=flip f
Prelude> :t g
g :: Double -> Double -> Double
Prelude> :t flip1 f
flip1 f :: (Fractional t1) => t1 -> t1 -> t1



The type of flip1 f is (Fractional t1) => t1 -> t1 -> t1, meaning that t1 is restricted to be of Fractional typeclass, but if I bind g to flip f, now the type is Double... that's not what one would expect! (Any Haskell expert can tell me what's happening?)


As usual, I'll tried to do the same in Scala to see how far it can goes.
The main difference is of course, type inference: in Scala you need to declare the function types. The other major difference is Haskell every function is curried by default, in Scala you need to explicitly curry the function by separating each parameter between parentheses.

In Scala, our (curried) f function will be:


def f(x:Int)(y:Int)=x/y
f: (Int)(Int)Int


The flip function in Scala would be:


def flip1[A,B,C]( f:A=>B=>C)(x:B)(y:A) =f(y)(x)
flip1: [A,B,C]((A) => (B) => C)(B)(A)C


Is clearly more verbose than the Haskell version, but equally powerful:


def g = flip1 (f) _
g: (Int) => (Int) => Int



The underscore is needed to get the partially applied function and the f must be in parentheses (again, all curried parameters must have their own parentheses).


Conclusion? Nothing. Haskell and Scala are obviously different, but they share some common ground... just wanted to show it :)

12/18/08

Playing with Scala 5 ½: More variations


As James pointed out, the implicit conversion creates a new object every time we convert the Node to Ordered[Node], and although the JVM and the GC can deal pretty well with that kind of scenario, it will be better if we can avoid the creation of the object (BTW, the fact that Scala can leverage years of work on JIT optimizations is something that shouldn't be taken lightly... I still want tail call optimization, though)

We need to add the Ordered trait for Node somewhere in the hierarchy. The reasonable place is when we add the Weight trait (once we have weight we can compare the nodes):

class DfsNode extends NodeImpl with Weight with Ordered[DfsNode]{

...

def compare(y: Node): Int = this.weight.compare(y.weight)

}

Now we can compare the nodes by their weight: “lighter” nodes are before “heavier” nodes, but the comparation is not what we need for the priority queue, we need to reverse the the ordering.

The most immediate way is just reverse DfsNode's comparation function:

def compare(y: Node): Int = -this.weight.compare(y.weight)

but this has the undesirable effect of reversing all the node comparations, and most of the time I'll bet we want to have the lighter nodes before the heavier ones, the reverse is counter-intuitive (and the potential source of nightmares). The solution? Let's create our particular flavor of DfsNode with the comparation reversed to use it in the PriorityQeue:

class PQNode extends DfsNode {

override def compare(y: Node): Int = -super.compare(y)

}

And we just need to change our newNode method to return PQNodes instead of DfsNodes:

protected def newNode: Node = new PQNode

For extra points on modularization, we can extract those changes and the code related to the algorithm in a subclass of DfsGraph.

Here is the complete code:




package dfs2;

abstract class Graph {
type Edge
type Node <: NodeIntf
abstract class NodeIntf {
def connectWith(node: Node): Edge
}
def nodes: Set[Node]
def edges: Set[Edge]
def addNode: Node
}

abstract class DirectedGraph extends Graph {
type Edge <: EdgeImpl
class EdgeImpl(origin: Node, dest: Node) {
def from = origin
def to = dest
}

class NodeImpl extends NodeIntf {
self: Node =>
def connectWith(node: Node): Edge = {
val edge = newEdge(this, node)
edges = edges + edge
edge
}
}

protected def newNode: Node
protected def newEdge(from: Node, to: Node): Edge

var nodes: Set[Node] =Set()
var edges: Set[Edge] =Set()
def addNode: Node = {
val node = newNode
nodes = nodes + node
node
}
}

trait Weight {
var weight= Float.PositiveInfinity
}

import scala.collection.mutable.PriorityQueue;

class DfsGraph extends DirectedGraph {
class DfsNode extends NodeImpl with Weight with Ordered[DfsNode]{
var previous: DfsNode = _
var label:String = _
var nodeEdges: Set[Edge] = Set()

override def connectWith(node: Node): Edge = {
val newEdge=super.connectWith(node)
nodeEdges = nodeEdges + newEdge
newEdge
}
def -->(n2:Node):DfsEdge =connectWith(n2)

override def toString()= label+ " w:"+weight
def compare(y: Node): Int = this.weight.compare(y.weight)
}


class DfsEdge(origin:Node, dest:Node) extends EdgeImpl(origin, dest) with Weight {
override def toString()= from.label+"-->"+to.label+" w:"+weight
}

def addNewNode(l:String):Node={
val newNode=super.addNode
newNode.label=l
newNode
}

type Node= DfsNode
type Edge= DfsEdge

protected def newEdge(from: Node, to: Node): Edge = new DfsEdge(from,to)
protected def newNode: Node = new DfsNode

}

class DijkstraSPGraph extends DfsGraph {

class PQNode extends DfsNode {
override def compare(y: Node): Int = -super.compare(y)
}

override protected def newNode: Node = new PQNode

def shortestPath(start: DfsNode, end: DfsNode) = {
val unvisited=new PriorityQueue[Node]()
unvisited++=(nodes)
start.weight=0
while (!unvisited.isEmpty) {
val vertx=unvisited.dequeue
for (v <- vertx.nodeEdges if canImprove(v)){
unvisited+improveDistance(v)
}
}
}

def canImprove(a:Edge)=(a.from.weight+a.weight< a.to.weight)

def improveDistance(a:Edge) ={
val v=a.to
v.weight=a.from.weight+a.weight
v.previous=a.from
v
}

def pathToStart(end:DfsNode):List[DfsNode] = {
if (end == null)
Nil
else
end :: pathToStart(end.previous)
}
}


object Example extends Application {

val graph = new DijkstraSPGraph

val n1=graph addNewNode "start"
val n2=graph addNewNode "n2"
val n3=graph addNewNode "n3"
val n4=graph addNewNode "n4"
val n5=graph addNewNode "n5"
val n6=graph addNewNode "end"
graph addNewNode "alone"

n1-->n2 weight=2
n1-->n3 weight=1
n2-->n4 weight=1
n3-->n4 weight=3
n2-->n5 weight=1
n4-->n6 weight=1
n5-->n6 weight=3

n1-->n6 weight=7
n1-->n4 weight=10

graph.shortestPath(n1,n6)
println("Path")
graph.pathToStart(n6).reverse.map(println(_))

}

12/14/08

Playing with Scala 5: improving code, use of views and comprehensions

After reading about different optimization to pathfinding algorithms, I decided to improve my Dijkstra implementation by using a priority queue instead of a set for the unvisited nodes. Using the “weight” of the vertex as the priority, we can avoid searching the whole set for the minimum and we reduce the cost of the operation from O(n) to O(log n).

The first change is replace the set with Scala's priority queue:


val unvisited=new PriorityQueue[Node]()




But as it is, it will not compile: the priority queue's signature is


class PriorityQueue[A](implicit view$1 : (A) => Ordered[A]) extends ResizableArray[A] with CloneableCollection


You need to provide an implicit conversion from the parameter type A to the type Ordered[A]. But nothing to worry about: is as easy as


implicit def node2ordered(n: Node): Ordered[Node] = new Ordered[Node] {

//reverse the comparation so highest priority = lower cost

def compare(y: Node): Int = -(n.weight.compare(y.weight))

}


Look at the improveDistance function, it does two things: filters the nodes and updates the distance if we can improve it. I think is better if we split into two different functions, separating the filtering from the updating, so instead of:


def improveDistance(a:Edge) ={
if (a.from.weight+a.weight< a.to.weight) {
a.to.weight=a.from.weight+a.weight
a.to.previous=a.from
}
}


we get:

def canImprove(a:Edge)=(a.from.weight+a.weight< a.to.weight)

and

def improveDistance(a:Edge) ={

a.to.weight=a.from.weight+a.weight

a.to.previous=a.from

a.to

}


Then, instead of a function map over the nodeEdges, we have to filter and then map:


vertx.nodeEdges.map(improveDistance(_))


is replaced by


vertx.nodeEdges.filter(canImprove(_))).map(improveDistance(_)


that gives a better picture of the steps involved.

As we need to add the updated nodes to the priority queue so they can “float” to the top, the line would look like:


unvisited++ (vertx.nodeEdges.filter(canImprove(_))).map(improveDistance(_))


But I don't really think the explicit use of map and filter helps readability.

Enter Scala's for comprehensions: we can replace the map/filter with a comprehension:


for (v <- vertx.nodeEdges if canImprove(v)){

unvisited+improveDistance(v)

}

Nothing fantastic, but an improvement, isn't it?

For future posts, I'm planning on generalizing the algorithm into a A* pathfinding with different heuristic functions... stay tuned!

10/22/08

Simple Java tricks to protect your web application against SQL injection

Your application is vulnerable to SQL Injection when you send unfiltered strings to the database. Most modern ORM frameworks should take care of it (but don't take my word!... go ahead and check how secure your framework is).
Sometimes, you have to work with plain JDBC (or ODBC). Here is a couple of tricks that help:
  1. First and foremost, avoid concatenating strings for SQL queries. Use prepared statements unless is not possible (i.e. cases when you have undefined number of parameters)
  2. Leverage the language type system: If you're passing a number, use Integer instead of String... any invalid character will fail the conversion and will not reach the DB.
  3. If there's no option but concatenate strings, make sure the database comment quotes are escaped (for example, in DB2 you have to replace the single quote character with 2 single quote characters: instead of "SELECT * FROM users WHERE name='"+param+"'" use "SELECT * FROM users WHERE name='"+param.replaceAll("'","''")+"'"
For something a little more advanced, you can wrap the strings in some kind of "EscapedString" class, and use that class in the signature of the DAOs (related to 2. )

Note: by no means this is a comprehensive list. Application security is very hard, check your database documentation...