14 September 2016

Transposing vectors of complex numbers in Matlab

Careful when applying the transpose (apostophe) operator to a vector of complex numbers. See here:

You have to use .' (dot transpose)
If you use the tranpose operator on a complex-valued vector, matrix in MATLAB, it returns the conjugate transpose by default.

01 September 2016

Correlation vs Convolution

This is a basic relationship in DSP.

When we talk about similarity between signals, we normally talk about "correlation".
When we talk about filtering, in the time domain that's a "convolution".

Correlation and convolution are basically the same thing.
The only difference is that correlation is a convolution by the inverted (or flipped) version of the same signal in the time domain:

y(n) = h(n)*x(n) --> convolution
y(n)=h(n)*x(-n) -> correlation

In Matlab/Octave that would be:

y = conv( h, x) for convolution

or

y = conv( h, flip(x)) for correlation.

In the frequency domain, the operation of flipping in the time domain translates into the "conjugate" version of the transform, that is with an inverted sign for its imaginary part.

02 August 2016

Fix for FR_NO_FILESYSTEM error with the embedded FatFS file system

FatFs is a popular file system for embedded applications.
Occasionally though I was having trouble accessing some SD cards, a problem that's been reported here.and here.
I've looked at a working SD card and a not working one.
They were different models and size but the problem came down to the second card not having its logical partition defined as primary. Using Minitool Partition Wizard you can fix this by doing:

1. Right Click on First Partition
2. Select Set Partition as Primary
2. Click on Apply Changes

The partition goes from Logical to Primary.

FAT32 also works, I've tried up to 8GB.

23 January 2014

How to reduce memory usage when marshalling large KML files with JAK (the Java API for KML)

This is something I discovered a while ago and never got around to publish it on this blog.

for a while I've been experiencing memory issues when marshalling large files.
I've monitored this usage with some crude profiling:
25,000 locations: 72MB
50,000 locations: 140MB

So I've looked for ways to reduce this problem. One approach is to marshal the file in chunks instead of all at once. Here are some useful links:

Example:

``JAXBContext context = JAXBContext.newInstance(type);``
``Marshaller` m = context.createMarshaller();`
````m.setProperty(Marshaller.JAXB_FRAGMENT, Boolean.TRUE);`
```
java.io.StringWriter sw = new java.io.StringWriter();
XMLStreamWriter xmlOut = XMLOutputFactory.newFactory().createXMLStreamWriter(sw);

xmlOut.writeStartDocument("UTF-8", "1.0");
xmlOut.writeStartElement("kml");
xmlOut.writeDefaultNamespace("http://www.opengis.net/kml/2.2");

xmlOut.writeNamespace("atom", "http://www.w3.org/2005/Atom");
xmlOut.writeNamespace("kml", "http://www.opengis.net/kml/2.2");
xmlOut.writeNamespace("xal", "urn:oasis:names:tc:ciq:xsdschema:xAL:2.0");

xmlOut.writeStartElement("Document");

// iterate through your placemarks here
Placemark placemark = new Placemark()
...
m.marshal(placemark, xmlOut);

xmlOut.writeEndElement(); // Document
xmlOut.writeEndElement(); // kml
xmlOut.close();

This is an intermediate solution that sacrifices elegance but this way I've been able to reduce memory usage at least by 60%:

25,000 locations: 20MB
50,000 locations: 40MB

I believe a similar approach can be used when parsing large documents.
I hope someone finds this useful

30 July 2013

My journey from Python to Java, to Scala... and back to Java

Here's an update on my work for improving the Construct library in java and make it more type safe and easy to use
https://github.com/ZiglioNZ/construct

Previous versions would work like this.
1. define a Struct at runtime, and composite it by passing in static constructors of other Constructs:

Struct s = Struct( "struct",
UBInt8("a"),
UBInt16("b"),
Struct("foo",
UBInt8("c"),
UBInt8("d")))

2. then we can use this struct to parse a byte array. That returns a Map type called Container:
Container c = struct.parse(ByteArray(1, 0, 2, 3, 4));

3. we can then extract the parsed value from the map/container:
assertEquals( 1,  c.get("a"));
assertEquals( 2,  c.get("b"));
assertEquals( 3, ((Container)c.get("foo")).get("c"));
assertEquals( 4, ((Container)c.get("foo")).get("d"));

You can see there are a few problems that make this API not so nice to use:
first, the problem with maps and key strings, one has to remember the names and refactoring for those names is painful
second, all that casting: due to java's lack of type inference, it's difficult to deal with HMaps, of the type that parsers return, therefore casting has to be added.

As a way to mitigate these problems, I've looked at Scala.
Scala offers nice things like elegant default constructor and case classes. Those two things combined could help creating a CLASS, instead of a runtime collection of fields.
Having a class would help the IDE with code completion: no longer I would have to remember field names, they would be class fields, therefore IDE code completion and refactoring.

I've thought long and hard and tried different things, and stumbled against some limitations of scala, case classes and inheritance.
At the end I went back to using Java reflection, with the idea that later I will look into Scala macros to improve it.

How have I done it?
Well, first instead of passing objects to a Struct at runtime, I statically define a Struct with a number of fields:

class Foo extends Struct {
public Foo(String name ){super(name);}
public UBInt8 c;
public UBInt8 d;
}

class S extends Struct {
public UBInt8 a;
public UBInt16 b;
public Foo foo;
}

Then I've added code to the Stuct constructor that at runtime uses reflection to inspect the Struct fields and create an instance of it, by passing the name of the field, that is also inspected via reflection.

The last trick is a way for each field to hold an Object, that is the result of a call to parse(). The Struct itself updates this value for each field, after parsing.
Now I have a get() method that returns that value for each field, see:

S s = new S();
s.parse(ByteArray(1, 0, 2, 3, 4));

assertEquals(1, s.a.get());
assertEquals(2, s.b.get());
assertEquals(3, s.foo.c.get());
assertEquals(4, s.foo.d.get());

I think it's much nicer to use. It's not type safe yet, since I need to update all my field definition in order to return the correct type, but I'm definitely getting there.

I think this is a win for java, that is still kicking.
can definitely explain all the subtleties why scala reflection would be better than java's, and why macros would be even better.
From the practical point of view I think this is a good compromise,

https://github.com/ZiglioNZ/construct/blob/83c01e14e30f7c04875f303cd423eecd14d97834/src/main/java/com/sirtrack/construct/Core.java

https://github.com/ZiglioNZ/construct/blob/83c01e14e30f7c04875f303cd423eecd14d97834/src/test/java/com/sirtrack/construct/ConstructTest.java

14 March 2013

Google Reading is shutting down 1 July 2013. What are (cloud based) alternatives?

Promising
Both products promise seamless integration with Reader, that is the ability to import all existing feeds. Both services are experiencing an obvious surge in traffic and are unable at the moment to accept new users.
Newsblur looks like it has good client support, web and mobile. Also it is an open source project.
From the FAQ thought it looks like their web service might not be able to poll regularly all the newsfeeds, but mainly the ones that people request the most.
I'd like to try it, and I'll be more than happy to pay for a good service.

Google's move also means definitive death for Listen, the podcast catcher I've been using all these years on Android.
Well, there are other solutions I know, but will they integrate with either newsblur or feedly?

[UPDATE, 2 April]
The latest Android update for gReader Pro claims "gReader will work after the closure. We are working on a solution." Fingers crossed

23 February 2013

Loving Open Source

Pretty much every component of our system is made out of open source components I can and have changed/contributed to.
Not every one of these components has a large community behind but sometimes I'm surprised by Pull Requests and Q&A on google groups or stack overflow.
These days sometimes it feels like I'm not growing as much as I used to, I don't learn entirely new things every day anymore. But looking back, gosh! the progress over the years has been constant and the accumulate knowledge significant. More than anything: written software still appears sound. If I go back, and I look at how certain things got implemented, even smaller things, they look simple and beautiful (not everything of course).
As a proof of that, extending and providing new services doesn't take too long.
Iterative refactoring allows me to consolidate the good things and to get rid of the bad (copy&paste) ones.

I'd like to take a chance and thank all the contributors to these projects:
Siena Project
Objectify
Construct
Open Layers
SlickGrid
KnockoutJS
Vosao CMS
Force.com Web Service Connector