23 February 2013

Loving Open Source

Pretty much every component of our system is made out of open source components I can and have changed/contributed to.
Not every one of these components has a large community behind but sometimes I'm surprised by Pull Requests and Q&A on google groups or stack overflow.
I find it invaluable having direct access to people who have implemented a library, sharing ideas, asking questions about their design.
These days sometimes it feels like I'm not growing as much as I used to, I don't learn entirely new things every day anymore. But looking back, gosh! the progress over the years has been constant and the accumulate knowledge significant. More than anything: written software still appears sound. If I go back, and I look at how certain things got implemented, even smaller things, they look simple and beautiful (not everything of course).
As a proof of that, extending and providing new services doesn't take too long.
Iterative refactoring allows me to consolidate the good things and to get rid of the bad (copy&paste) ones.

I'd like to take a chance and thank all the contributors to these projects:
Siena Project
Objectify
Construct
Open Layers
SlickGrid
KnockoutJS
Vosao CMS
Force.com Web Service Connector
All Google's Contributions and Resources!

plus several others...
For my contributions in my free time and at work go to:

github.com/ZiglioNZ/
github.com/sirtrack

PS Recently I've been invited to take part of a project related to my latest interest: strabismus and amblyopia.
There's no public information available for this particular project yet but it will eventually become Open Source on GitHub!

17 July 2012

Java Construct 1.1.2 Release

Java Construct 1.1.2 is now available!

You can download it from https://github.com/ZiglioUK/construct/tags

Also available from Maven Central

 <dependency>  
  <groupId>com.sirtrack</groupId>  
  <artifactId>javaconstruct</artifactId>  
  <version>1.1.2</version>  
 </dependency>

New features:

a CRC/Checksum Construct, see example on how to use it: IPv4
Implemented Restreams (dynamic Structs) - only parsing for now
Implemented MetaArray, Sequence, more Adapters and Macros
Added an experimental BeanAdapter for typesafe conversion from/to a Container

TODOs:

Improve formatting. I'm sorry but each new release of Eclipse loses my settings. It looks good on Eclipse, but for anyone else tabs are all over the place
Implement Restream.build() at a certain point

Nice to have:

Do some performance testing, using the Jvm Serializers benchmark
Cleanup the low level, possibly moving away from ByteBuffer, in similar way as Kryo 2 did.

About Construct

Python Construct is a library for parsing and building binary messages.

Java Construct is an "as faithful as possible" translation to Java of Python Construct 2.x.

See the docs for explanations and examples.

20 January 2012

Java Construct 1.0.0 Release

I'm pleased to announce release 1.0.0 of Java Construct, a faithful
port to Java of Python Construct.

About Python Construct: http://construct.wikispaces.com/
"Construct is a python library for parsing and building of data
structures (binary or textual). It is based on the concept of defining
data structures in a declarative manner, rather than procedural code:
more complex constructs are composed of a hierarchy of simpler ones.
It's the first library that makes parsing fun, instead of the usual
headache it is today."

About Java Construct: https://github.com/ZiglioUK/construct/
This Java version employs some syntactic sugar (i.e. static methods)
to make the syntax as close as possible to the original Construct
library in Python.

Example of a Construct:

 import static construct.Core.*;  
   import static construct.Macros.*;  
   import static construct.Adapters.*;  
   import static construct.lib.Containers.*;  
   Construct struct = BitStruct(  
     "foo",  
     BitField("a", 3),  
     Flag("b"),  
     Padding(3),  
     Nibble("c"),  
     Struct("bar",  
       Nibble("d"),  
       Bit("e")  
     )  
   );  

A Java Construct can parse byte arrays and produces Objects like
Containers. Viceversa, it can take Objects to produce byte arrays.

   public Object parse(byte[] data);  
   public byte[] build( Object obj);  

Parsing example:

   Container c1 = Container(  
     "a", 7,  
     "b", false,  
     "bar",  
     Container(  
       "d", 15 ,  
       "e", 1  
      ),  
      "c",8  
   );  
   Container c2 = struct.parse( ByteArray( 0xe1, 0x1f ));  
   assertEquals( c1, c2 );  

Currently Java Construct supports enough Macros, Adapters and
Repeaters to parse and build these protocols:

Layer 2: ethernet, arparp
Layer 3: ipv4, ipv6
Layer 4: udp, tcptcp
ipstack



Full ipstack example: https://github.com/ZiglioUK/construct/blob/master/src/main/construct/protocols/ipstack.java

Notes:
1. I haven't tested for threadsafety but it's a priority
2. Streams are not supported so a message has to be contained in
memory. If there are segments, they have to re-assambled prior to
parsing
3. Text protocols like http are not supported, it's questionable
whether Construct would be the right tool for text parsing.

14 November 2011

Using MapReduce to refactor entities on GAE

I've been following Ikai Lan's good tutorial for MapReduce on App Engine (Java):
http://ikaisays.com/2010/07/09/using-the-java-mapper-framework-for-app-engine/

As a simple excercise I've managed to succesfully refactor some entities on the production server.
The goal is to take existing rows of an existing entity called Event and rename a field from Program to Project.
On the App Engine dashboard I can query for them with this GQL String:
"SELECT * FROM Event WHERE program != null".
At the end of this procedure, the GQL query shouldn't return any row.

Here are four simple steps.

1. Edit mapreduce.xml where I specify my mapper class and pass the Entity kind 'Event' as a parameter

 <configurations>  
  <configuration name="Program to Project">  
   <property>  
    <name>mapreduce.map.class</name>  
    <value>com.sirtrack.iridium.mapper.ProgramToProject</value>  
   </property>  
   <property>  
    <name>mapreduce.inputformat.class</name>  
    <value>com.google.appengine.tools.mapreduce.DatastoreInputFormat</value>  
   </property>  
   <!--property>  
    <name human="Entity Kind to Map Over">mapreduce.mapper.inputformat.datastoreinputformat.entitykind</name>  
    <value template="optional">Event</value>  
   </property-->  
   <property>  
    <name human="Entity Kind to Map Over">mapreduce.mapper.inputformat.datastoreinputformat.entitykind</name>  
    <value>Event</value>  
   </property>  
  </configuration>  
 </configurations>

2. Implement the mapper class

 package com.sirtrack.iridium.mapper;  
 import com.google.appengine.api.datastore.DatastoreService;  
 import com.google.appengine.api.datastore.DatastoreServiceFactory;  
 import com.google.appengine.api.datastore.Entity;  
 import com.google.appengine.api.datastore.Key;  
 import com.google.appengine.tools.mapreduce.AppEngineMapper;  
 import com.googlecode.objectify.cache.CachingDatastoreServiceFactory;  
 import org.apache.hadoop.io.NullWritable;  
 import java.util.logging.Logger;  
 public class ProgramToProject extends AppEngineMapper< Key,Entity,NullWritable,NullWritable>  
 {  
  private static final Logger log = Logger.getLogger( ProgramToProject.class.getName() );  
  private DatastoreService datastore;  
  public ProgramToProject()  
  {  
  }  
  @Override  
  public void taskSetup( Context context )  
  {  
   this.datastore = DatastoreServiceFactory.getDatastoreService();  
  }  
  @Override  
  public void map( Key key, Entity value, Context context )  
  {  
   log.warning( "Mapping key: " + key );  
   if( value.hasProperty( "program" ) )  
   {  
    Object program = value.getProperty( "program" );  
    value.setProperty( "project", program );  
    value.setProperty( "program", null );  
    datastore.put( value );  
   }  
  }  
 }

3. Add a link to the MapReduce admin page to the dashboard (appengine-web.xml). In my case, I've mapped the mapreduce serverlet to /_ah/mapreduce, a protected address

  <admin-console>  
   <page name="Appstats" url="/_ah/appstats" />  
   <page name="Mapreduce" url="/_ah/mapreduce/status" />  
  </admin-console>

4. Run the job from the dashboard

Program to Project

Job #job_1321220718997aeac4fdbfc3f4a7cba174e4f28845724_0001

Processed items per shard

Overview

DONE
Elapsed time: 00:00:23
Start time: Mon Nov 14 2011 10:45:19 GMT+1300 (NZDT)

Counters

org.apache.hadoop.mapred.Task$Counter:MAP_INPUT_RECORDS: 983 (42.21/sec avg.)

Done! I can now verify that it all worked using the same GQL query I used before.

Conclusion and Future Improvements

MapReduce is a great tool for GAE, and also very easy to use but I'd like to use my favourite API (Siena) instead of the low level access to the datastore. It shouldn't be difficult.
There's also a more efficient way of performing this task, also covered by Ikai's tutorial.

19 December 2010

'Google Listen' for Podcasts on Android

Last week the Google Reader app for Android was released.
To be honest, I've been disappointed. I find the application really slow, and that may be due to a number of reasons:
- too many feeds (hundreds for me)
- slow connection (3G not always available)
- uncompressed XML on Android 2.1

Also, I've been experiencing an annoying bug: items that appear as 'read' on my mobile are in reality still unread. Sometimes, after spending some time on my mobile going through all items, when I connect to Reader I find them all again as 'unread'. Strangely enough items I've 'starred' on Android seem to maintain that state but not always. I wonder whether that might related to me having multiple accounts.

I've been using Reader for a long time on the desktop also as a sort of iTune for podcasts. I've got a number, not large but very active, of subscriptions that I keep in a special folder within Reader.
While that works fairly well on the desktop, it doesn't suit very well as podcast catcher on mobile. That's mainly because when you click on a podcast in Reader, that starts streaming but it's not saved anywhere in particular. Basically you either listen to it or you'll have to stream it later, there's no way in Reader to automatically save the podcast for later use.

Until now! I've tried 'Google Listen' as a suggested application by App Brain.
I had previously installed another Podcast application but never got to use it.
The main feature that attracted me to Listen was its integration with Google Reader.

How does that work?

Well, there are a few ways to enter a subscription in Listen:
1. by hand
2. by clicking on an RSS link
3. as a newly created folder in Reader

The first way is really painful, RSS urls are usually long and tiresome to type on a small screen.
The second way, I haven't tried yet but it involves, I presume, visiting a web page with a link to a podcast.
The third way is brilliant in my case because it just meant moving my existing subscriptions in Reader from my old Podcast folder to the new 'Listen Subscriptions' folder.

As an application Listen allows you to do all the expected: create a list of podcasts to download, set when they will be downloaded, etc.
It's also way faster and more pleasant to use than Reader itself on Android.

Still there's room for improvements, I'd like to be able to:
1. share podcasts to Facebook, Twitter, E-mail etc
2. automatically set an item as read, same as in Reader
3, star an item and prevent it from being deleted
4. save an item to a preferred location

Google Listen for Android is a great little app, it'll become the one I use the most.

18 September 2010

App Inventor for Android

Just received this e-mail from Google

Welcome to App Inventor!

About App Inventor:
App Inventor for Android allows people with minimal programming experience to create simple, personal apps for Android devices. It has a number of features which ease app development. App Inventor is best suited for people who are eager to learn the basics of programming and are interested in making basic apps for their personal use.

Remember the Beta Tag:
App Inventor is currently available as an invitation-based beta product. We are limiting access in order to ensure that our systems can handle the load. As a beta product App Inventor still has rough edges and missing features. In some cases the rough edges include un-pleasantries such as: installing Windows device drivers, installing Java on your computer, and fiddling with settings on your phone. We are working hard to smooth out the rough edges and we appreciate your use of App Inventor while we are in this beta state. It may sound a bit cliche but it really is true, your use of App Inventor today will help us make it better for the future! Now on to the good stuff.

Getting Started with App Inventor:

Complete these 3 steps to start using App Inventor:
Set up your phone and computer.

Connect your phone to your computer.

Complete the basic tutorials.

Take your App Inventor knowledge to the next level by:
Working through the advanced tutorials.

Reading the App Inventor reference material.

Subscribing to the App Inventor user forum.

Happy Inventing!
Google's App Inventor Team

05 September 2010

Keeping track of changes with Google Reader

Recently I heard JavaPosse's Dick Wall stating: "Google Reader is my lifeline".

I agree! More and more content is user generated. But not all content is quality and most of it still comes from traditional sites.

That's where a feed aggregator like Google Reader can help.

A feed aggregator is a software that collates syndicated content called RSS feed. That is a machine readable file that reports parts or all the content of a page. When available, a special icon usually appears somewhere on the page or on the browser's url field.

If you click on that icon you'll be taken to the Url of the Rss feed. The page appears as scrambled text. It is in reality an XML representation of the page you were looking at. You can grab that Url and paste into a feed aggregator such as Google Reader.
The feed aggregator will start polling that Url and will prompt you when the page is updated.

Home
All items (2)
Starred items

Your stuff

Shared items

Notes

Trends

Browse for stuff
People you follow

The initial effort of copying and pasting that link soon pays off: you don't have to go and visit web sites you follow regularly in order to check updates. The updates come to you soon after they're published via the feed reader.
That way I've been able to monitor much more interesting stuff than I could have done by hand.

Not all Rss feeds reports the whole content of the page is linked to. Often news sites provide just an excerpt from their stories and a link back to the stories themselves. That's for obvious reasons: generating traffic to their ad hosting pages, fair enough.

Not all syndicated content is interesting or original. A lot of times, the same news comes from different sources. That's where Google Reader's Key Shortcuts are really useful.

If you click '?' on Google Reader, a list of those shortcuts will appear.

The ones I use a lot are:

- g+a: go to new posts

- n: skip to the next post

At a certain point it may become too much, with hundreds of new feeds being collected every day. It's then time to decide what sites is worth following and what's not.

Google Reader, being web based, has a series of advantages with respect to stand-alone readers:

it's connected to your gmail or google apps account, so it's always updated and available wherever you connect from. That's useful when you follow a mix of work and personal sites
you can search across your subscriptions to find interesting stuff you've read before
with time you build a profile of what you're interested in and Google Reader can suggest you sources you might be interested in.

But there's another neat feature. Google Reader allows you track changes of any web page, even those that don't offer Rss! If you enter the location of a page without Rss, Reader will start polling that page and over time it will automatically build a summary of changes.

That's useful for pages like 'job opportunities' or 'current exhibitions' at museums such as: http://www.pataka.org.nz/48977/links/bulletpages.html

Google feed for "http://www.pataka.org.nz/48977/links/bulletpages.html"
Google will watch for changes in "http://www.pataka.org.nz/48977/links/bulletpages.html" and summarize them for you.

If I wanted to follow the news about the recent earthquake in Christchurch, I could go to Google News and simply enter the keyword 'Christchurch'.

At the end of the result page, Google offers a way to receive the updates via e-mail.

But who wants to get tens of news alerts in their mailboxes. The e-mail is for personal messages that often need a reply!

There's a much better way to keep track of updates: the Rss feed.

Search for the

RSS icon at the bottom of the news results. You can copy and paste that link into Google Reader, a much better way!

25 June 2010

I'm human!

Google

Sorry...

We're sorry...

... but your computer or network may be sending automated queries. To protect our users, we can't process your request right now.

See Google Help for more information.

11 June 2010

Phitek CEO Donaldson pushed out

http://www.nzherald.co.nz/business/news/article.cfm?c_id=3&objectid=10651158

That's quite rude... Engineers are not necessarily good CEOs. I hope Mark finds another role, or another idea!