I'm pleased to announce release 1.0.0 of Java Construct, a faithful port to Java of Python Construct. About Python Construct: http://construct.wikispaces.com/ "Construct is a python library for parsing and building of data structures (binary or textual). It is based on the concept of defining data structures in a declarative manner, rather than procedural code: more complex constructs are composed of a hierarchy of simpler ones. It's the first library that makes parsing fun, instead of the usual headache it is today." About Java Construct: https://github.com/ZiglioUK/construct/ This Java version employs some syntactic sugar (i.e. static methods) to make the syntax as close as possible to the original Construct library in Python. Example of a Construct:A Java Construct can parse byte arrays and produces Objects like Containers. Viceversa, it can take Objects to produce byte arrays.import static construct.Core.*; import static construct.Macros.*; import static construct.Adapters.*; import static construct.lib.Containers.*; Construct struct = BitStruct( "foo", BitField("a", 3), Flag("b"), Padding(3), Nibble("c"), Struct("bar", Nibble("d"), Bit("e") ) );Parsing example:public Object parse(byte[] data); public byte[] build( Object obj);Currently Java Construct supports enough Macros, Adapters and Repeaters to parse and build these protocols: Full ipstack example: https://github.com/ZiglioUK/construct/blob/master/src/main/construct/protocols/ipstack.java Notes: 1. I haven't tested for threadsafety but it's a priority 2. Streams are not supported so a message has to be contained in memory. If there are segments, they have to re-assambled prior to parsing 3. Text protocols like http are not supported, it's questionable whether Construct would be the right tool for text parsing.Container c1 = Container( "a", 7, "b", false, "bar", Container( "d", 15 , "e", 1 ), "c",8 ); Container c2 = struct.parse( ByteArray( 0xe1, 0x1f )); assertEquals( c1, c2 );
20 January 2012
Java Construct 1.0.0 Release
14 November 2011
Using MapReduce to refactor entities on GAE
I've been following Ikai Lan's good tutorial for MapReduce on App Engine (Java):
http://ikaisays.com/2010/07/09/using-the-java-mapper-framework-for-app-engine/
As a simple excercise I've managed to succesfully refactor some entities on the production server.
The goal is to take existing rows of an existing entity called Event and rename a field from Program to Project.
On the App Engine dashboard I can query for them with this GQL String:
"SELECT * FROM Event WHERE program != null".
At the end of this procedure, the GQL query shouldn't return any row.
Here are four simple steps.
1. Edit mapreduce.xml where I specify my mapper class and pass the Entity kind 'Event' as a parameter
2. Implement the mapper class
3. Add a link to the MapReduce admin page to the dashboard (appengine-web.xml). In my case, I've mapped the mapreduce serverlet to /_ah/mapreduce, a protected address
4. Run the job from the dashboard
There's also a more efficient way of performing this task, also covered by Ikai's tutorial.
http://ikaisays.com/2010/07/09/using-the-java-mapper-framework-for-app-engine/
As a simple excercise I've managed to succesfully refactor some entities on the production server.
The goal is to take existing rows of an existing entity called Event and rename a field from Program to Project.
On the App Engine dashboard I can query for them with this GQL String:
"SELECT * FROM Event WHERE program != null".
At the end of this procedure, the GQL query shouldn't return any row.
Here are four simple steps.
1. Edit mapreduce.xml where I specify my mapper class and pass the Entity kind 'Event' as a parameter
<configurations>
<configuration name="Program to Project">
<property>
<name>mapreduce.map.class</name>
<value>com.sirtrack.iridium.mapper.ProgramToProject</value>
</property>
<property>
<name>mapreduce.inputformat.class</name>
<value>com.google.appengine.tools.mapreduce.DatastoreInputFormat</value>
</property>
<!--property>
<name human="Entity Kind to Map Over">mapreduce.mapper.inputformat.datastoreinputformat.entitykind</name>
<value template="optional">Event</value>
</property-->
<property>
<name human="Entity Kind to Map Over">mapreduce.mapper.inputformat.datastoreinputformat.entitykind</name>
<value>Event</value>
</property>
</configuration>
</configurations>
2. Implement the mapper class
package com.sirtrack.iridium.mapper;
import com.google.appengine.api.datastore.DatastoreService;
import com.google.appengine.api.datastore.DatastoreServiceFactory;
import com.google.appengine.api.datastore.Entity;
import com.google.appengine.api.datastore.Key;
import com.google.appengine.tools.mapreduce.AppEngineMapper;
import com.googlecode.objectify.cache.CachingDatastoreServiceFactory;
import org.apache.hadoop.io.NullWritable;
import java.util.logging.Logger;
public class ProgramToProject extends AppEngineMapper< Key,Entity,NullWritable,NullWritable>
{
private static final Logger log = Logger.getLogger( ProgramToProject.class.getName() );
private DatastoreService datastore;
public ProgramToProject()
{
}
@Override
public void taskSetup( Context context )
{
this.datastore = DatastoreServiceFactory.getDatastoreService();
}
@Override
public void map( Key key, Entity value, Context context )
{
log.warning( "Mapping key: " + key );
if( value.hasProperty( "program" ) )
{
Object program = value.getProperty( "program" );
value.setProperty( "project", program );
value.setProperty( "program", null );
datastore.put( value );
}
}
}
3. Add a link to the MapReduce admin page to the dashboard (appengine-web.xml). In my case, I've mapped the mapreduce serverlet to /_ah/mapreduce, a protected address
<admin-console>
<page name="Appstats" url="/_ah/appstats" />
<page name="Mapreduce" url="/_ah/mapreduce/status" />
</admin-console>
4. Run the job from the dashboard
Program to Project
Job #job_1321220718997aeac4fdbfc3f4a7cba174e4f28845724_0001
Processed items per shard
Overview
- DONE
- Elapsed time: 00:00:23
- Start time: Mon Nov 14 2011 10:45:19 GMT+1300 (NZDT)
Counters
- org.apache.hadoop.mapred.Task$Counter:MAP_INPUT_RECORDS: 983 (42.21/sec avg.)
Done! I can now verify that it all worked using the same GQL query I used before.
Conclusion and Future Improvements
MapReduce is a great tool for GAE, and also very easy to use but I'd like to use my favourite API (Siena) instead of the low level access to the datastore. It shouldn't be difficult.Conclusion and Future Improvements
There's also a more efficient way of performing this task, also covered by Ikai's tutorial.
19 December 2010
'Google Listen' for Podcasts on Android
Last week the Google Reader app for Android was released.
To be honest, I've been disappointed. I find the application really slow, and that may be due to a number of reasons:
- too many feeds (hundreds for me)
- slow connection (3G not always available)
- uncompressed XML on Android 2.1
Also, I've been experiencing an annoying bug: items that appear as 'read' on my mobile are in reality still unread. Sometimes, after spending some time on my mobile going through all items, when I connect to Reader I find them all again as 'unread'. Strangely enough items I've 'starred' on Android seem to maintain that state but not always. I wonder whether that might related to me having multiple accounts.
I've been using Reader for a long time on the desktop also as a sort of iTune for podcasts. I've got a number, not large but very active, of subscriptions that I keep in a special folder within Reader.
While that works fairly well on the desktop, it doesn't suit very well as podcast catcher on mobile. That's mainly because when you click on a podcast in Reader, that starts streaming but it's not saved anywhere in particular. Basically you either listen to it or you'll have to stream it later, there's no way in Reader to automatically save the podcast for later use.
Until now! I've tried 'Google Listen' as a suggested application by App Brain.
I had previously installed another Podcast application but never got to use it.
The main feature that attracted me to Listen was its integration with Google Reader.
How does that work?
Well, there are a few ways to enter a subscription in Listen:
1. by hand
2. by clicking on an RSS link
3. as a newly created folder in Reader
The first way is really painful, RSS urls are usually long and tiresome to type on a small screen.
The second way, I haven't tried yet but it involves, I presume, visiting a web page with a link to a podcast.
The third way is brilliant in my case because it just meant moving my existing subscriptions in Reader from my old Podcast folder to the new 'Listen Subscriptions' folder.
As an application Listen allows you to do all the expected: create a list of podcasts to download, set when they will be downloaded, etc.
It's also way faster and more pleasant to use than Reader itself on Android.
Still there's room for improvements, I'd like to be able to:
1. share podcasts to Facebook, Twitter, E-mail etc
2. automatically set an item as read, same as in Reader
3, star an item and prevent it from being deleted
4. save an item to a preferred location
Google Listen for Android is a great little app, it'll become the one I use the most.
18 September 2010
App Inventor for Android
Just received this e-mail from Google
Welcome to App Inventor!
Remember the Beta Tag:
App Inventor is currently available as an invitation-based beta product. We are limiting access in order to ensure that our systems can handle the load. As a beta product App Inventor still has rough edges and missing features. In some cases the rough edges include un-pleasantries such as: installing Windows device drivers, installing Java on your computer, and fiddling with settings on your phone. We are working hard to smooth out the rough edges and we appreciate your use of App Inventor while we are in this beta state. It may sound a bit cliche but it really is true, your use of App Inventor today will help us make it better for the future! Now on to the good stuff.
Getting Started with App Inventor:
Complete these 3 steps to start using App Inventor:
- Set up your phone and computer.
- Connect your phone to your computer.
- Complete the basic tutorials.
Take your App Inventor knowledge to the next level by:
- Working through the advanced tutorials.
- Reading the App Inventor reference material.
- Subscribing to the App Inventor user forum.
Happy Inventing!Google's App Inventor Team
05 September 2010
Keeping track of changes with Google Reader
Recently I heard JavaPosse's Dick Wall stating: "Google Reader is my lifeline".
But there's another neat feature. Google Reader allows you track changes of any web page, even those that don't offer Rss! If you enter the location of a page without Rss, Reader will start polling that page and over time it will automatically build a summary of changes.
I agree! More and more content is user generated. But not all content is quality and most of it still comes from traditional sites.
That's where a feed aggregator like Google Reader can help.
A feed aggregator is a software that collates syndicated content called RSS feed. That is a machine readable file that reports parts or all the content of a page. When available, a special icon usually appears somewhere on the page or on the browser's url field.
If you click on that icon you'll be taken to the Url of the Rss feed. The page appears as scrambled text. It is in reality an XML representation of the page you were looking at. You can grab that Url and paste into a feed aggregator such as Google Reader.
The feed aggregator will start polling that Url and will prompt you when the page is updated.
The initial effort of copying and pasting that link soon pays off: you don't have to go and visit web sites you follow regularly in order to check updates. The updates come to you soon after they're published via the feed reader.
That way I've been able to monitor much more interesting stuff than I could have done by hand.
A feed aggregator is a software that collates syndicated content called RSS feed. That is a machine readable file that reports parts or all the content of a page. When available, a special icon usually appears somewhere on the page or on the browser's url field.
If you click on that icon you'll be taken to the Url of the Rss feed. The page appears as scrambled text. It is in reality an XML representation of the page you were looking at. You can grab that Url and paste into a feed aggregator such as Google Reader.The feed aggregator will start polling that Url and will prompt you when the page is updated.
The initial effort of copying and pasting that link soon pays off: you don't have to go and visit web sites you follow regularly in order to check updates. The updates come to you soon after they're published via the feed reader.
That way I've been able to monitor much more interesting stuff than I could have done by hand.
Not all Rss feeds reports the whole content of the page is linked to. Often news sites provide just an excerpt from their stories and a link back to the stories themselves. That's for obvious reasons: generating traffic to their ad hosting pages, fair enough.
Not all syndicated content is interesting or original. A lot of times, the same news comes from different sources. That's where Google Reader's Key Shortcuts are really useful.
If you click '?' on Google Reader, a list of those shortcuts will appear.
The ones I use a lot are:
- g+a: go to new posts
- n: skip to the next post
At a certain point it may become too much, with hundreds of new feeds being collected every day. It's then time to decide what sites is worth following and what's not.
Google Reader, being web based, has a series of advantages with respect to stand-alone readers:
- it's connected to your gmail or google apps account, so it's always updated and available wherever you connect from. That's useful when you follow a mix of work and personal sites
- you can search across your subscriptions to find interesting stuff you've read before
- with time you build a profile of what you're interested in and Google Reader can suggest you sources you might be interested in.
But there's another neat feature. Google Reader allows you track changes of any web page, even those that don't offer Rss! If you enter the location of a page without Rss, Reader will start polling that page and over time it will automatically build a summary of changes.
That's useful for pages like 'job opportunities' or 'current exhibitions' at museums such as: http://www.pataka.org.nz/48977/links/bulletpages.html
Google feed for "http://www.pataka.org.nz/48977/links/bulletpages.html"
Google will watch for changes in "http://www.pataka.org.nz/48977/links/bulletpages.html" and summarize them for you.
If I wanted to follow the news about the recent earthquake in Christchurch, I could go to Google News and simply enter the keyword 'Christchurch'.
At the end of the result page, Google offers a way to receive the updates via e-mail.
But who wants to get tens of news alerts in their mailboxes. The e-mail is for personal messages that often need a reply!
There's a much better way to keep track of updates: the Rss feed.
Search for the 
RSS icon at the bottom of the news results. You can copy and paste that link into Google Reader, a much better way!
RSS icon at the bottom of the news results. You can copy and paste that link into Google Reader, a much better way!25 June 2010
I'm human!
Sorry... |
We're sorry...
... but your computer or network may be sending automated queries. To protect our users, we can't process your request right now.See Google Help for more information.
© 2010 Google - Google Home
11 June 2010
Phitek CEO Donaldson pushed out
http://www.nzherald.co.nz/business/news/article.cfm?c_id=3&objectid=10651158
That's quite rude... Engineers are not necessarily good CEOs. I hope Mark finds another role, or another idea!
That's quite rude... Engineers are not necessarily good CEOs. I hope Mark finds another role, or another idea!
10 June 2010
"
08 June 2010
Anywhere will be better than here!
22°C
Current: Mostly Cloudy
Wind: N at 8 km/h
Humidity: 73%
Wind: N at 8 km/h
Humidity: 73%
Tue
27° | 18°
Wed
27° | 18°
Thu
23° | 17°
Fri
25° | 17°
28°C
Current: Light rain
Wind: N at 0 km/h
Humidity: 79%
Wind: N at 0 km/h
Humidity: 79%
Tue
34° | 26°
Wed
31° | 25°
Thu
34° | 25°
Fri
33° | 26°
4°C
Wind: S at 10 km/h
Humidity: 82%
Tue
9° | 3°
Wed
10° | 3°
Thu
13° | 6°
Fri
15° | 7°
26°C
Current: Rain Showers
Humidity: 84%
Humidity: 84%
Wed
29° | 25°
Thu
28° | 25°
Fri
29° | 26°
Sat
30° | 26°
香港天文台 6月8日17時2分發布
22°C
Current: Mostly Cloudy
Wind: SE at 11 km/h
Humidity: 60%
Wind: SE at 11 km/h
Humidity: 60%
Tue
24° | 14°
Wed
29° | 17°
Thu
32° | 21°
Fri
27° | 15°
21°C
Current: Cloudy
Wind: N at 6 km/h
Humidity: 49%
Wind: N at 6 km/h
Humidity: 49%
Tue
25° | 16°
Wed
29° | 17°
Thu
33° | 21°
Fri
28° | 17°
31 May 2010
Google Power Meter
An article on stuff presents a device made by Current Cost that monitors energy consumption in real time and sends data usage to Google Power Meter.
A user though has to keep a computer running all the time in order to collect data wirelessly from the device and update it to Google.
Through Powershop, that is also possible today. I've personally spent a few nights working with Powershop's developer API and managed so far to collect data from Powershop and then send it to Google.
Check this thread out: http://groups.google.com/group/powershop-developers/browse_thread/thread/54460039ec71fa1b
My aim is to develop a Google App Engine application that could collect that for me at a set interval.
Powershop's Ari Sargent has also experimented with Google Powermeter and they may provide a link with that tool themselves.
By the way, their web site is excellent and they could provide similar, or better, gadgets.
14 May 2010
ASB has blocked all credit card electronic transactions to Italy (this week)
I tried a couple of days ago to purchase a domain from a reputable Italian provider and for the first time it was denied. I managed do it via Paypal though.
Today ASB rang me to say that credit card transactions to the whole italian country are blocked, as a (hopefully) temporary measure. Here's their e-mail:
Today ASB rang me to say that credit card transactions to the whole italian country are blocked, as a (hopefully) temporary measure. Here's their e-mail:
Recently you attempted a transaction at an Italian merchant. This transaction was declined. Unfortunately we have had to take action to prevent any and all transactions from being processed at any Italian merchant where the card is not present. This is a result of recent fraud activity and while we understand that this has inconvenienced you, and we apologise for this inconvenience, we feel this is necessary to protect our customers by preventing any possible fraud. Should you be travelling to Italy your card will be accepted where you present it at the merchant just not available for internet, mail order or phone transactions.
Subscribe to:
Posts (Atom)