Embedded Flight Controller Tutorial

This is an excellent series. I'll post the links here because I keep losing them:

• Project 1: Custom Flight Controller Part 0: Overview
• Custom Flight Controller Part 1: A Simple Revision on Quadcopter Flying Principle
• Custom Flight Controller Part 2.1: Getting the Orientation --- Get Data from MPU9250 Sensor Module 
• Custom Flight Controller Part 2.2: Getting the Orientation --- The Quaternion Kalman Filter
• Custom Flight Controller Part 2.3: Getting the Orientation --- The Matlab Simulation 
• Custom Flight Controller Part 2.4: Getting the Orientation --- Migrate to STM32 
• Custom Flight Controller Part 2.5: Getting the Orientation --- QKF on STM32 
• Custom Flight Controller Part 2.6: Getting the Orientation --- Using Interrupt to Get Data from MPU9250 
• Custom Flight Controller Part 2.7: Getting the Orientation --- Mag Calibration and Overall Algorithm Structure. 
• Custom Flight Controller Part 2.8: Getting the Orientation --- Processing Orientation Visualization

And here's a link to the code on GitHub.

Thank you Lyfkyle, great work!

Using Java 8's Function to refactor Decorator and Adapter patterns

Two days into using Java 8 (I know, long overdue, but you know... App Engine), I'm having fun refactoring some legacy (Swing, ouch) codebase.

One of the most powerful new tools is "Function", and I'm discovering how you can refactor entire classes, anymous or not, with one line of code, see for example this question:

Is it possible to pass parameters, or access external parameters to an anonymous class? For example:

int myVariable = 1;

myButton.addActionListener(new ActionListener() {
    public void actionPerformed(ActionEvent e) {
        // How would one access myVariable here?
    }
});

Here's my proposed solution, using Function:

Function<Integer,ActionListener> printInt = 
    intvar -> ae -> System.out.println(intvar);

int myVariable = 1;
myButton.addActionListener( printInt.apply(myVariable) );

I think the power of Function becomes obvious when we apply it to refactoring patters such as Decorator, Adapter or other type of proxies.

Here's some code I've just refactored, showing how you can refactor an entire class with a one liner Function:

PS Don't miss this talk by Trisha Gee

Transposing vectors of complex numbers in Matlab

Careful when applying the transpose (apostophe) operator to a vector of complex numbers. See here:

You have to use .' (dot transpose)
If you use the tranpose operator on a complex-valued vector, matrix in MATLAB, it returns the conjugate transpose by default.

Correlation vs Convolution

This is a basic relationship in DSP.

When we talk about similarity between signals, we normally talk about "correlation".
When we talk about filtering, in the time domain that's a "convolution".

Correlation and convolution are basically the same thing.
The only difference is that correlation is a convolution by the inverted (or flipped) version of the same signal in the time domain:

y(n) = h(n)*x(n) --> convolution
y(n)=h(n)*x(-n) -> correlation

In Matlab/Octave that would be:

y = conv( h, x) for convolution

or

y = conv( h, flip(x)) for correlation.

In the frequency domain, the operation of flipping in the time domain translates into the "conjugate" version of the transform, that is with an inverted sign for its imaginary part.

Fix for FR_NO_FILESYSTEM error with the embedded FatFS file system

FatFs is a popular file system for embedded applications.
Occasionally though I was having trouble accessing some SD cards, a problem that's been reported here.and here.
I've looked at a working SD card and a not working one.
They were different models and size but the problem came down to the second card not having its logical partition defined as primary. Using Minitool Partition Wizard you can fix this by doing:

1. Right Click on First Partition
2. Select Set Partition as Primary
2. Click on Apply Changes

The partition goes from Logical to Primary.

FAT32 also works, I've tried up to 8GB.

How to reduce memory usage when marshalling large KML files with JAK (the Java API for KML)

This is something I discovered a while ago and never got around to publish it on this blog.

for a while I've been experiencing memory issues when marshalling large files.

I've monitored this usage with some crude profiling:

25,000 locations: 72MB

50,000 locations: 140MB

So I've looked for ways to reduce this problem. One approach is to marshal the file in chunks instead of all at once. Here are some useful links:

http://stackoverflow.com/questions/1775202/how-to-stream-large-files-using-jaxb-marshaller

http://rezarahim.blogspot.co.nz/2010/05/chunking-out-big-xml-with-stax-and-jaxb.html

Example:

JAXBContext context = JAXBContext.newInstance(type);

Marshaller m = context.createMarshaller();

m.setProperty(Marshaller.JAXB_FRAGMENT, Boolean.TRUE);

    java.io.StringWriter sw = new java.io.StringWriter();

    XMLStreamWriter xmlOut = XMLOutputFactory.newFactory().createXMLStreamWriter(sw);

  

    xmlOut.writeStartDocument("UTF-8", "1.0");

    xmlOut.writeStartElement("kml");

    xmlOut.writeDefaultNamespace("http://www.opengis.net/kml/2.2");

    xmlOut.writeNamespace("atom", "http://www.w3.org/2005/Atom");

    xmlOut.writeNamespace("kml", "http://www.opengis.net/kml/2.2");

    xmlOut.writeNamespace("gx", "http://www.google.com/kml/ext/2.2");

    xmlOut.writeNamespace("xal", "urn:oasis:names:tc:ciq:xsdschema:xAL:2.0");

        

    xmlOut.writeStartElement("Document");

  // iterate through your placemarks here

    Placemark placemark = new Placemark()

   ...

    m.marshal(placemark, xmlOut);

 

     xmlOut.writeEndElement(); // Document

     xmlOut.writeEndElement(); // kml

     xmlOut.close();

    

This is an intermediate solution that sacrifices elegance but this way I've been able to reduce memory usage at least by 60%:

25,000 locations: 20MB

50,000 locations: 40MB

I believe a similar approach can be used when parsing large documents.

I hope someone finds this useful

My journey from Python to Java, to Scala... and back to Java

Here's an update on my work for improving the Construct library in java and make it more type safe and easy to use 
https://github.com/ZiglioNZ/construct

Previous versions would work like this.
1. define a Struct at runtime, and composite it by passing in static constructors of other Constructs:

Struct s = Struct( "struct",
     UBInt8("a"),
     UBInt16("b"),
     Struct("foo",
           UBInt8("c"),
           UBInt8("d")))

2. then we can use this struct to parse a byte array. That returns a Map type called Container:
Container c = struct.parse(ByteArray(1, 0, 2, 3, 4));

3. we can then extract the parsed value from the map/container:
assertEquals( 1,  c.get("a"));
assertEquals( 2,  c.get("b")); 
assertEquals( 3, ((Container)c.get("foo")).get("c"));
assertEquals( 4, ((Container)c.get("foo")).get("d"));

You can see there are a few problems that make this API not so nice to use: 
first, the problem with maps and key strings, one has to remember the names and refactoring for those names is painful
second, all that casting: due to java's lack of type inference, it's difficult to deal with HMaps, of the type that parsers return, therefore casting has to be added.

As a way to mitigate these problems, I've looked at Scala.
Scala offers nice things like elegant default constructor and case classes. Those two things combined could help creating a CLASS, instead of a runtime collection of fields. 
Having a class would help the IDE with code completion: no longer I would have to remember field names, they would be class fields, therefore IDE code completion and refactoring.

I've thought long and hard and tried different things, and stumbled against some limitations of scala, case classes and inheritance.
At the end I went back to using Java reflection, with the idea that later I will look into Scala macros to improve it.

How have I done it?
Well, first instead of passing objects to a Struct at runtime, I statically define a Struct with a number of fields:

    class Foo extends Struct {
      public Foo(String name ){super(name);}
      public UBInt8 c;
      public UBInt8 d;
    }

    class S extends Struct {
      public UBInt8 a;
      public UBInt16 b;
      public Foo foo;
    }

Then I've added code to the Stuct constructor that at runtime uses reflection to inspect the Struct fields and create an instance of it, by passing the name of the field, that is also inspected via reflection.

The last trick is a way for each field to hold an Object, that is the result of a call to parse(). The Struct itself updates this value for each field, after parsing.
Now I have a get() method that returns that value for each field, see:

    S s = new S();
    s.parse(ByteArray(1, 0, 2, 3, 4));

    assertEquals(1, s.a.get());
    assertEquals(2, s.b.get());
    assertEquals(3, s.foo.c.get());
    assertEquals(4, s.foo.d.get());

I think it's much nicer to use. It's not type safe yet, since I need to update all my field definition in order to return the correct type, but I'm definitely getting there.

I think this is a win for java, that is still kicking.
+Pascal Voitot Dev can definitely explain all the subtleties why scala reflection would be better than java's, and why macros would be even better. 
From the practical point of view I think this is a good compromise,


https://github.com/ZiglioNZ/construct/blob/83c01e14e30f7c04875f303cd423eecd14d97834/src/main/java/com/sirtrack/construct/Core.java

https://github.com/ZiglioNZ/construct/blob/83c01e14e30f7c04875f303cd423eecd14d97834/src/test/java/com/sirtrack/construct/ConstructTest.java

Google Reader Alternatives

Google Reading is shutting down 1 July 2013. What are (cloud based) alternatives?

Promising

1. http://www.newsblur.com/
2. http://feedly.com/

Both products promise seamless integration with Reader, that is the ability to import all existing feeds. Both services are experiencing an obvious surge in traffic and are unable at the moment to accept new users.

Newsblur looks like it has good client support, web and mobile. Also it is an open source project.

From the FAQ thought it looks like their web service might not be able to poll regularly all the newsfeeds, but mainly the ones that people request the most.

I'd like to try it, and I'll be more than happy to pay for a good service.

Google's move also means definitive death for Listen, the podcast catcher I've been using all these years on Android.

Well, there are other solutions I know, but will they integrate with either newsblur or feedly?

[UPDATE, 2 April]
The latest Android update for gReader Pro claims "gReader will work after the closure. We are working on a solution." Fingers crossed

Loving Open Source

Pretty much every component of our system is made out of open source components I can and have changed/contributed to.
Not every one of these components has a large community behind but sometimes I'm surprised by Pull Requests and Q&A on google groups or stack overflow.
I find it invaluable having direct access to people who have implemented a library, sharing ideas, asking questions about their design.
These days sometimes it feels like I'm not growing as much as I used to, I don't learn entirely new things every day anymore. But looking back, gosh! the progress over the years has been constant and the accumulate knowledge significant. More than anything: written software still appears sound. If I go back, and I look at how certain things got implemented, even smaller things, they look simple and beautiful (not everything of course).
As a proof of that, extending and providing new services doesn't take too long. 
Iterative refactoring allows me to consolidate the good things and to get rid of the bad (copy&paste) ones.

I'd like to take a chance and thank all the contributors to these projects:
Siena Project
Objectify
Construct
Open Layers
SlickGrid
KnockoutJS
Vosao CMS
Force.com Web Service Connector
All Google's Contributions and Resources!


plus several others... 
For my contributions in my free time and at work go to:

github.com/ZiglioNZ/
github.com/sirtrack



PS Recently I've been invited to take part of a project related to my latest interest: strabismus and amblyopia.
There's no public information available for this particular project yet but it will eventually become Open Source on GitHub!

Java Construct 1.1.2 Release

Java Construct 1.1.2 is now available!

You can download it from https://github.com/ZiglioUK/construct/tags

Also available from Maven Central

 <dependency>  
  <groupId>com.sirtrack</groupId>  
  <artifactId>javaconstruct</artifactId>  
  <version>1.1.2</version>  
 </dependency>  

New features:

1. a CRC/Checksum Construct, see example on how to use it: IPv4 
2. Implemented Restreams (dynamic Structs) - only parsing for now
3. Implemented MetaArray, Sequence, more Adapters and Macros
4. Added an experimental BeanAdapter for typesafe conversion from/to a Container

TODOs:

1. Improve formatting. I'm sorry but each new release of Eclipse loses my settings. It looks good on Eclipse, but for anyone else tabs are all over the place
2. Implement Restream.build() at a certain point 

Nice to have:

1. Do some performance testing, using the Jvm Serializers benchmark
2. Cleanup the low level, possibly moving away from ByteBuffer, in similar way as Kryo 2 did.

About Construct

Python Construct is a library for parsing and building binary messages. 

Java Construct is an "as faithful as possible" translation to Java of Python Construct 2.x. 

See the docs for explanations and examples. 

Java Construct 1.0.0 Release

I'm pleased to announce release 1.0.0 of Java Construct, a faithful
port to Java of Python Construct.

About Python Construct: http://construct.wikispaces.com/
"Construct is a python library for parsing and building of data
structures (binary or textual). It is based on the concept of defining
data structures in a declarative manner, rather than procedural code:
more complex constructs are composed of a hierarchy of simpler ones.
It's the first library that makes parsing fun, instead of the usual
headache it is today."

About Java Construct: https://github.com/ZiglioUK/construct/
This Java version employs some syntactic sugar (i.e. static methods)
to make the syntax as close as possible to the original Construct
library in Python.

Example of a Construct:

 import static construct.Core.*;  
   import static construct.Macros.*;  
   import static construct.Adapters.*;  
   import static construct.lib.Containers.*;  
   Construct struct = BitStruct(  
     "foo",  
     BitField("a", 3),  
     Flag("b"),  
     Padding(3),  
     Nibble("c"),  
     Struct("bar",  
       Nibble("d"),  
       Bit("e")  
     )  
   );  

A Java Construct can parse byte arrays and produces Objects like
Containers. Viceversa, it can take Objects to produce byte arrays.

   public Object parse(byte[] data);  
   public byte[] build( Object obj);  

Parsing example:

   Container c1 = Container(  
     "a", 7,  
     "b", false,  
     "bar",  
     Container(  
       "d", 15 ,  
       "e", 1  
      ),  
      "c",8  
   );  
   Container c2 = struct.parse( ByteArray( 0xe1, 0x1f ));  
   assertEquals( c1, c2 );  

Currently Java Construct supports enough Macros, Adapters and
Repeaters to parse and build these protocols:

Layer 2: ethernet, arparp
Layer 3: ipv4, ipv6
Layer 4: udp, tcptcp
ipstack



Full ipstack example: https://github.com/ZiglioUK/construct/blob/master/src/main/construct/protocols/ipstack.java

Notes:
1. I haven't tested for threadsafety but it's a priority
2. Streams are not supported so a message has to be contained in
memory. If there are segments, they have to re-assambled prior to
parsing
3. Text protocols like http are not supported, it's questionable
whether Construct would be the right tool for text parsing.

Using MapReduce to refactor entities on GAE

I've been following Ikai Lan's good tutorial for MapReduce on App Engine (Java):
http://ikaisays.com/2010/07/09/using-the-java-mapper-framework-for-app-engine/

As a simple excercise I've managed to succesfully refactor some entities on the production server.
The goal is to take existing rows of an existing entity called Event and rename a field from Program to Project.
On the App Engine dashboard I can query for them with this GQL String:
"SELECT * FROM Event WHERE program != null".
At the end of this procedure, the GQL query shouldn't return any row.

Here are four simple steps.

1. Edit mapreduce.xml where I specify my mapper class and pass the Entity kind 'Event' as a parameter

 <configurations>  
  <configuration name="Program to Project">  
   <property>  
    <name>mapreduce.map.class</name>  
    <value>com.sirtrack.iridium.mapper.ProgramToProject</value>  
   </property>  
   <property>  
    <name>mapreduce.inputformat.class</name>  
    <value>com.google.appengine.tools.mapreduce.DatastoreInputFormat</value>  
   </property>  
   <!--property>  
    <name human="Entity Kind to Map Over">mapreduce.mapper.inputformat.datastoreinputformat.entitykind</name>  
    <value template="optional">Event</value>  
   </property-->  
   <property>  
    <name human="Entity Kind to Map Over">mapreduce.mapper.inputformat.datastoreinputformat.entitykind</name>  
    <value>Event</value>  
   </property>  
  </configuration>  
 </configurations>  

2. Implement the mapper class

 package com.sirtrack.iridium.mapper;  
 import com.google.appengine.api.datastore.DatastoreService;  
 import com.google.appengine.api.datastore.DatastoreServiceFactory;  
 import com.google.appengine.api.datastore.Entity;  
 import com.google.appengine.api.datastore.Key;  
 import com.google.appengine.tools.mapreduce.AppEngineMapper;  
 import com.googlecode.objectify.cache.CachingDatastoreServiceFactory;  
 import org.apache.hadoop.io.NullWritable;  
 import java.util.logging.Logger;  
 public class ProgramToProject extends AppEngineMapper< Key,Entity,NullWritable,NullWritable>  
 {  
  private static final Logger log = Logger.getLogger( ProgramToProject.class.getName() );  
  private DatastoreService datastore;  
  public ProgramToProject()  
  {  
  }  
  @Override  
  public void taskSetup( Context context )  
  {  
   this.datastore = DatastoreServiceFactory.getDatastoreService();  
  }  
  @Override  
  public void map( Key key, Entity value, Context context )  
  {  
   log.warning( "Mapping key: " + key );  
   if( value.hasProperty( "program" ) )  
   {  
    Object program = value.getProperty( "program" );  
    value.setProperty( "project", program );  
    value.setProperty( "program", null );  
    datastore.put( value );  
   }  
  }  
 }  

3. Add a link to the MapReduce admin page to the dashboard (appengine-web.xml). In my case, I've mapped the mapreduce serverlet to /_ah/mapreduce, a protected address

  <admin-console>  
   <page name="Appstats" url="/_ah/appstats" />  
   <page name="Mapreduce" url="/_ah/mapreduce/status" />  
  </admin-console>  

4. Run the job from the dashboard

Program to Project

Job #job_1321220718997aeac4fdbfc3f4a7cba174e4f28845724_0001

Processed items per shard

Overview

• DONE
• Elapsed time: 00:00:23
• Start time: Mon Nov 14 2011 10:45:19 GMT+1300 (NZDT)

Counters

• org.apache.hadoop.mapred.Task$Counter:MAP_INPUT_RECORDS: 983 (42.21/sec avg.)

Done! I can now verify that it all worked using the same GQL query I used before.


Conclusion and Future Improvements

MapReduce is a great tool for GAE, and also very easy to use but I'd like to use my favourite API (Siena) instead of the low level access to the datastore. It shouldn't be difficult.
There's also a more efficient way of performing this task, also covered by Ikai's tutorial.

'Google Listen' for Podcasts on Android

Last week the Google Reader app for Android was released.
To be honest, I've been disappointed. I find the application really slow, and that may be due to a number of reasons:
- too many feeds (hundreds for me)
- slow connection (3G not always available)
- uncompressed XML on Android 2.1

Also, I've been experiencing an annoying bug: items that appear as 'read' on my mobile are in reality still unread. Sometimes, after spending some time on my mobile going through all items, when I connect to Reader I find them all again as 'unread'. Strangely enough items I've 'starred' on Android seem to maintain that state but not always. I wonder whether that might related to me having multiple accounts.

I've been using Reader for a long time on the desktop also as a sort of iTune for podcasts. I've got a number, not large but very active, of subscriptions that I keep in a special folder within Reader.
While that works fairly well on the desktop, it doesn't suit very well as podcast catcher on mobile. That's mainly because when you click on a podcast in Reader, that starts streaming but it's not saved anywhere in particular. Basically you either listen to it or you'll have to stream it later, there's no way in Reader to automatically save the podcast for later use.

Until now! I've tried 'Google Listen' as a suggested application by App Brain.
I had previously installed another Podcast application but never got to use it.
The main feature that attracted me to Listen was its integration with Google Reader.

How does that work?

Well, there are a few ways to enter a subscription in Listen:
1. by hand
2. by clicking on an RSS link
3. as a newly created folder in Reader

The first way is really painful, RSS urls are usually long and tiresome to type on a small screen.
The second way, I haven't tried yet but it involves, I presume, visiting a web page with a link to a podcast.
The third way is brilliant in my case because it just meant moving my existing subscriptions in Reader from my old Podcast folder to the new 'Listen Subscriptions' folder.

As an application Listen allows you to do all the expected: create a list of podcasts to download, set when they will be downloaded, etc.
It's also way faster and more pleasant to use than Reader itself on Android.

Still there's room for improvements, I'd like to be able to:
1. share podcasts to Facebook, Twitter, E-mail etc
2. automatically set an item as read, same as in Reader
3, star an item and prevent it from being deleted
4. save an item to a preferred location

Google Listen for Android is a great little app, it'll become the one I use the most.

App Inventor for Android

Just received this e-mail from Google

Welcome to App Inventor!

About App Inventor: App Inventor for Android allows people with minimal programming experience to create simple, personal apps for Android devices. It has a number of features which ease app development. App Inventor is best suited for people who are eager to learn the basics of programming and are interested in making basic apps for their personal use. Remember the Beta Tag: App Inventor is currently available as an invitation-based beta product. We are limiting access in order to ensure that our systems can handle the load. As a beta product App Inventor still has rough edges and missing features. In some cases the rough edges include un-pleasantries such as: installing Windows device drivers, installing Java on your computer, and fiddling with settings on your phone. We are working hard to smooth out the rough edges and we appreciate your use of App Inventor while we are in this beta state. It may sound a bit cliche but it really is true, your use of App Inventor today will help us make it better for the future! Now on to the good stuff. Getting Started with App Inventor: Complete these 3 steps to start using App Inventor: Set up your phone and computer. Connect your phone to your computer. Complete the basic tutorials. Take your App Inventor knowledge to the next level by: Working through the advanced tutorials. Reading the App Inventor reference material. Subscribing to the App Inventor user forum. Happy Inventing! Google's App Inventor Team

Keeping track of changes with Google Reader

Recently I heard JavaPosse's Dick Wall stating: "Google Reader is my lifeline".

I agree! More and more content is user generated. But not all content is quality and most of it still comes from traditional sites. 

That's where a feed aggregator like Google Reader can help.

A feed aggregator is a software that collates syndicated content called RSS feed. That is a machine readable file that reports parts or all the content of a page. When available,  a special icon usually appears somewhere on the page or on the browser's url field.
If you click on that icon you'll be taken to the Url of the Rss feed. The page appears as scrambled text. It is in reality an XML representation of the page you were looking at. You can grab that Url and paste into a feed aggregator such as Google Reader.
The feed aggregator will start polling that Url and will prompt you when the page is updated.

Home

All items (2)

Starred items

• 
  
  Your stuff
  
  • Shared items
  • Notes

Trends

Browse for stuff

• People you follow

The initial effort of copying and pasting that link soon pays off: you don't have to go and visit web sites you follow regularly in order to check updates. The updates come to you soon after they're published via the feed reader.
That way I've been able to monitor much more interesting stuff than I could have done by hand.

Not all Rss feeds reports the whole content of the page is linked to. Often news sites provide just an excerpt from their stories and a link back to the stories themselves. That's for obvious reasons: generating traffic to their ad hosting pages, fair enough.

Not all syndicated content is interesting or original. A lot of times, the same news comes from different sources. That's where Google Reader's Key Shortcuts are really useful.

If you click '?' on Google Reader, a list of those shortcuts will appear.

The ones I use a lot are:

- g+a: go to new posts

- n: skip to the next post

At a certain point it may become too much, with hundreds of new feeds being collected every day. It's then time to decide what sites is worth following and what's not.

Google Reader, being web based, has a series of advantages with respect to stand-alone readers:

• it's connected to your gmail or google apps account, so it's always updated and available wherever you connect from. That's useful when you follow a mix of work and personal sites
• you can search across your subscriptions to find interesting stuff you've read before
• with time you build a profile of what you're interested in and Google Reader can suggest you sources you might be interested in. 

But there's another neat feature. Google Reader allows you track changes of any web page, even those that don't offer Rss! If you enter the location of a page without Rss, Reader will start polling that page and over time it will automatically build a summary of changes.

That's useful for pages like 'job opportunities' or 'current exhibitions' at museums such as: http://www.pataka.org.nz/48977/links/bulletpages.html

Google feed for "http://www.pataka.org.nz/48977/links/bulletpages.html"

Google will watch for changes in "http://www.pataka.org.nz/48977/links/bulletpages.html" and summarize them for you.

If I wanted to follow the news about the recent earthquake in Christchurch, I could go to Google News and simply enter the keyword 'Christchurch'. 

At the end of the result page, Google offers a way to receive the updates via e-mail.

But who wants to get tens of news alerts in their mailboxes. The e-mail is for personal messages that often need a reply! 

There's a much better way to keep track of updates: the Rss feed.

Search for the RSS icon at the bottom of the news results. You can copy and paste that link into Google Reader, a much better way!