30 July 2013

My journey from Python to Java, to Scala... and back to Java

Here's an update on my work for improving the Construct library in java and make it more type safe and easy to use 
https://github.com/ZiglioNZ/construct

Previous versions would work like this.
1. define a Struct at runtime, and composite it by passing in static constructors of other Constructs:

Struct s = Struct( "struct",
     UBInt8("a"),
     UBInt16("b"),
     Struct("foo",
           UBInt8("c"),
           UBInt8("d")))

2. then we can use this struct to parse a byte array. That returns a Map type called Container:
Container c = struct.parse(ByteArray(1, 0, 2, 3, 4));

3. we can then extract the parsed value from the map/container:
assertEquals( 1,  c.get("a"));
assertEquals( 2,  c.get("b")); 
assertEquals( 3, ((Container)c.get("foo")).get("c"));
assertEquals( 4, ((Container)c.get("foo")).get("d"));

You can see there are a few problems that make this API not so nice to use: 
first, the problem with maps and key strings, one has to remember the names and refactoring for those names is painful
second, all that casting: due to java's lack of type inference, it's difficult to deal with HMaps, of the type that parsers return, therefore casting has to be added.

As a way to mitigate these problems, I've looked at Scala.
Scala offers nice things like elegant default constructor and case classes. Those two things combined could help creating a CLASS, instead of a runtime collection of fields. 
Having a class would help the IDE with code completion: no longer I would have to remember field names, they would be class fields, therefore IDE code completion and refactoring.

I've thought long and hard and tried different things, and stumbled against some limitations of scala, case classes and inheritance.
At the end I went back to using Java reflection, with the idea that later I will look into Scala macros to improve it.

How have I done it?
Well, first instead of passing objects to a Struct at runtime, I statically define a Struct with a number of fields:

    class Foo extends Struct {
      public Foo(String name ){super(name);}
      public UBInt8 c;
      public UBInt8 d;
    }

    class S extends Struct {
      public UBInt8 a;
      public UBInt16 b;
      public Foo foo;
    }

Then I've added code to the Stuct constructor that at runtime uses reflection to inspect the Struct fields and create an instance of it, by passing the name of the field, that is also inspected via reflection.

The last trick is a way for each field to hold an Object, that is the result of a call to parse(). The Struct itself updates this value for each field, after parsing.
Now I have a get() method that returns that value for each field, see:

    S s = new S();
    s.parse(ByteArray(1, 0, 2, 3, 4));

    assertEquals(1, s.a.get());
    assertEquals(2, s.b.get());
    assertEquals(3, s.foo.c.get());
    assertEquals(4, s.foo.d.get());

I think it's much nicer to use. It's not type safe yet, since I need to update all my field definition in order to return the correct type, but I'm definitely getting there.

I think this is a win for java, that is still kicking.
+Pascal Voitot Dev can definitely explain all the subtleties why scala reflection would be better than java's, and why macros would be even better. 
From the practical point of view I think this is a good compromise,


https://github.com/ZiglioNZ/construct/blob/83c01e14e30f7c04875f303cd423eecd14d97834/src/main/java/com/sirtrack/construct/Core.java

https://github.com/ZiglioNZ/construct/blob/83c01e14e30f7c04875f303cd423eecd14d97834/src/test/java/com/sirtrack/construct/ConstructTest.java