Multigesture.net

A Deepdive Debugging WebP Image Support on Android

Wed, 06 Mar 2024 11:00:00 +0000

Whether it’s the app’s stability, a smoother user experience or better app performance (e.g. reduced loading times), we’re always looking for ways to improve our app at Whatnot. As an Android engineer on the Discovery team, I was tasked in late ’23 to review our feed-driven surfaces. This post focuses on auditing the network calls the app was making as well as looking at the bandwidth usage of the Android app. For a livestream shopping app where latency and performance sit at the core of our experience, this investment led to better app and scroll performance for our Android Whatnot users, which is especially important as Whatnot continues to grow internationally where there are many Android users.

Starting point

Android Studio comes with an inbuilt Network traffic inspector that allows you to review your network calls. For more granular information, we prefer to use a tool like Proxyman to capture and inspect network traffic with ease. Especially being able to filter out specific endpoints or inspect network requests originating from the third-party SDKs in your app.

Besides slimming down your network requests (e.g. request fewer fields in your API calls), the other thing that we can do is to ensure that we only request images that are appropriately sized for mobile devices.

The JPG format has been the de facto standard for the web for a while now and it is no surprise that we have been using this on Android as well. As new image formats were developed (e.g. WebP, HEIF, AVIF) we had options for alternative more efficient image formats. On the web, WebP is slowly gaining popularity and is supported by all major modern browsers. The main benefit of migrating from JPG to WebP is the reduction of the image file size while maintaining a reasonable picture quality.

With the help of Proxyman, we did a simple test. For the same show, we requested the show image but in different file formats: JPG, PNG and WebP.

jpg (68.1 KB) | png (877 KB) 😂 | webp (45.1 KB)

On average, we measured a reduction of ~34% when using WebP which seems to be on par with the WebP Compression Study.

A smaller file size, means faster image loading times, a more responsive app and hopefully happier users.

The importance of internal dogfooding

Regularly using our own products internally helps us catch bugs early. Initially, when we released the changes internally, things seemed to be ok until one of our community moderators noticed that something was wrong with some of his listings. They were showing up correctly on the profile shop on his iPhone, but not on his Android device. On the Android app, the WebP images were being displayed sideways. Luckily this problem was found early and we were able to revert the change so it wouldn’t impact our users.

Finding the culprit

With all the benefits of WebP, it would be a shame not to be able to use it on Android.

Not knowing what could be the reason for the wrong image orientation, the first step was to see if we could find some hints by checking the image data.

Similar to the JPG image format, WebP images can also contain an Exchangeable Image File Format (EXIF) section that stores information about the image such as creation date, capturing device, image size, image rotation, color profiles, GPS locations, etc.

As the images were displayed correctly on iOS it got us wondering if our image server was malforming the images during the conversion process to WebP or if the image library we are using on Android, Coil, had a bug in its implementation.

Inspecting the EXIF data

There are plenty of tools we can use to review the EXIF data of an image, two popular ones are Exiv2 and ExifTool.

The first thing we did was to create some test material to replicate the problem our community moderator was seeing. We used an iPhone to take photos in 4 different orientations. By default these images were captured as JPGs and were uploaded to our image server. Next, we requested a WebP conversion via our image service and saved them locally.

This allowed us to have 4 original images and 4 converted images that we could inspect. Using the ExifTool on both sets of files resulted in this:

File Name : case1-portait.jpg
Orientation : Rotate 90 CW

File Name : case1-portait.webp
Orientation : Rotate 90 CW

File Name : case2-landscape-turned-left.jpg
Orientation : Horizontal (normal)

File Name : case2-landscape-turned-left.webp
Orientation : Horizontal (normal)

File Name : case3-upside-down-portrait.jpg
Orientation : Rotate 270 CW

File Name : case3-upside-down-portrait.webp
Orientation : Rotate 270 CW

File Name : case4-landscape-turned-right.jpg
Orientation : Rotate 180

File Name : case4-landscape-turned-right.webp
Orientation : Rotate 180

Looking at the results meant that the original image (JPG) contained proper orientation information and that the converted image (WebP) also contained orientation information.

As a sanity check, we tried to open up the WebP files from the image service on Safari, and MacOS’s Finder which both displayed the image correctly. Chrome, Firefox and our app (using Coil) all seemed to fail which was very suspicious.

Diving deeper into Coil

As previously mentioned, we use a 3rd party image library called Coil to handle all our image loading in the app. So the next step would be to review if there is an issue with the library itself. Fortunately, Coil is open source so we have easy access to the code. Placing many breakpoints in the code and stepping through the image loader functions revealed that the orientation reported by Coil was off for some files.

While the correct orientation was reported for JPG files, it failed to show any orientation information for our WebP files. Very odd, could the problem be somewhere else?

The AndroidX Exifinterface

While Coil helps you load images into your Composables/Views, it relies on the AndroidX Exifinterface to read the EXIF data. Like Android, this is open source so that anyone can download and review (or fix) the source code.

To make debugging easier, we created a test application that would directly interface with the ExifInterface.

One thing we noticed while looking at the source code of the ExifInterface was that Google did cover a test case with a WebP image containing EXIF data. Why did that one work?

Let’s get low level

At this point, we know that something is up between the Google WebP file and our own. Before we start inspecting the files, it’s a good idea to look up the official specifications of the image data container format. In the case of WebP, Google has published them here.

Since we’re dealing with binary files, we won’t be able to use a text editor. Instead, we need something more advanced such as a hex editor.

A hex editor allows you to inspect binaries in hexadecimal as well as text. On macOS, we can use a free tool called Hex Fiend to open the two files.

Google’s WebP | Our WebP

Hmm, wait a moment, the EXIF structure looks a bit different between the two files. Let’s take a look at what the EXIF structure is supposed to look like in the specs.

There should be an EXIF chunk header and a payload with the metadata. In both cases, we can see that the “EXIF” marker (in plain text) is present.

The Exif metadata section should start with a so-called TIFF header. This header contains information about the byte order (Endianness) of the payload. The first two bytes are 0x49 0x49 (II for Intel) if the byte order is little-endian or 0x4D 0x4D (MM for Motorola) for big-endian. The following two bytes are magic bytes 0x00 0x2A (42 😉).

Going back to our files we can see that the file generated by our image service contains an additional section that looks like this: 0x45 0x78 0x69 0x66 0x00 0x00 (“Exif\0\0”) before the TIFF header but why?

The EXIF APP1 Section

Checking on Google, it appears that our WebP files include an additional APP1 marker — uhm, isn’t this supposed to be standardized?

Reviewing the EXIF standards by the Camera & Imaging Products Association (CIPA), it becomes clear that the APP1 is a JPG-specific marker that precedes the EXIF data (and is not considered part of the EXIF data payload).

It’s important to note that the official WebP specifications do not mention the APP1 marker at all, meaning WebP files should have never contained this APP1 marker in the first place.

It’s very possible that the bug originates from image processing libraries (and that affects any frameworks relying on them), for example:

This confirms that our image service is copying EXIF data directly from the original JPG file into the newly created WebP files.

Images in the wild

So here we are, there are images out there because some image libraries didn’t completely adhere to the specifications and malformed WebP files that are now floating around the internet.

Our best guess is that Safari, MacOS’s Finder, ExifTool and Exiv2 all use a relaxed EXIF metadata reader while Firefox, Chrome and the (AndroidX) ExifInterface are more strict.

Fixing the problem

Going back to the ExifInterface source code we were able to track down the problem. It was failing to read the TIFF header and that resulted in not reading any of the EXIF data.

Knowing this, we created a patch that would scan the first 6 bytes of the payload. If the 6 bytes match the JPG EXIF APP1 marker, we adjust the offset and skip the number of bytes and the length of the marker before processing the rest of the data.

To validate the changes we updated the test application

Unpatched ExifInterface | Patched ExifInterface

Success!

We provided the patch to Google which was merged into the codebase at https://android-review.googlesource.com/c/platform/frameworks/support/+/2889627

What’s next

It’s rare to stumble upon a bug this low level, but it’s always helpful to know your debugging tool options. While the debugger in Android Studio is very advanced and should cover most of your day-to-day app development needs, external tools like Proxyman and Hex Editors can make your life much easier when it’s time to debug a problem.

The complicated part here is that the bug is likely caused by an external image generation framework (seemingly harmless) but that didn’t surface until a more strict EXIF library was trying to read the data.

Now that the AndroidX ExifInterface is patched, libraries like Coil will also be able to benefit from this change.

Since we have full control of our image service, we also decided to circumvent the issue on the backend by reading EXIF data from the JPG file and applying the rotation on the requested image. The benefit of this approach is that it will now work out of the box on Android, Chrome and Firefox plus it allows us to strip away all EXIF information for privacy reasons.

_{This article was previous posted on the Whatnot Engineering blog}

My mobile journey

Sat, 25 Sep 2021 19:02:33 +0000

If you are a mobile developer and haven’t read Chet Haase’s latest book “Androids: The team that built the Android operating system” go read it now! It’s a delightful read and a wonderful trip into Android’s history!

Androids

The team that created Android didn’t just create a new mobile OS they also created a new career path for software engineers like me. I’ve been coding for Android since November 2008, which means almost 13 years! The projects that I’ve worked on have been very… diverse.

When the team announced their Android device, I wasn’t living in the US (yet) and they weren’t selling it in the Netherlands yet. I was intrigued by what it was, a smartphone or really a portable tiny computer. Fortunately, for work at that time, I was going to be in Austin, TX in Nov 2008 for SuperComputing 2008 to demo a multi-touch device. After the conference was over, I went to a T-Mobile store and tried to purchase a G1. Back then they didn’t like it when you would try to just buy the device without service, but with a bit of patience they eventually sold one to me.

The first app that I wrote wasn’t going to be “Hello World”, nah, I needed a bigger challenge so I tried porting a Chip 8 emulator that I had written for Windows sometime earlier. When I ported it to Java using Eclipse (yep before Android Studio was a thing), the first game I tried was Space Invaders. I was so excited that the code ran and I was able to play it with the hardware keyboard!

Running Space Invaders (via emulation) on my G1

At that time I was still active on EFnet and Freenode chat servers, I thought it would be cool to have an IRC client to check in every now and then from my phone. That’s when I started to work on fIRC which was released just before Christmas of 2008. I never expected so many people would download and use it. By default it would drop users in #android-chat on Freenode which became very popular and often time had hilarious chatter going on.

Running fIRC and mIRC side by side

The old version started with great ratings and high number of downloads… until it didn’t work so well on newer Android devices as I had a lot of layout things hardcoded for landscape use

Funny enough, fIRC was also the reason I got in touch with, San Mehat, one of the Android engineers who used to idle in the chatroom too (I think most Android engineers were on #android on Freenode back then). As fIRC was gaining traction, at some point there was so much talk in different languages going on that it became hard to follow any conversations. At one point he yelled “WILL YOU MORONS SPEAK ENGLISH?” and that’s how a new IRC topic was born.

In the spring of 2009, I was still doing research work on multi-touch tech and was invited to come to the Interactive Displays Conference in San Jose. As I had never been in the area before (and it was my first trip to the US by myself), I was asking for some ideas on IRC in #android-chat on things to do in the SF area (as I knew a bunch of them where SF locals). San saw the message and said that we should meet up! We actually did end up meeting in person… (what was I thinking? meeting up with a stranger from the internet??). San is a great guy, even though I was just starting out as a software developer, we had a great chat and geeked out a bit over the G1. I still remember he offered me to come to the Google Mountain View campus but unfortunately I was there only for about a week and the schedule didn’t allow for it. I still regret that.

Late 2011 I started freelancing via Epic Windmill and one of my friends (Seth Sandler) asked me if I could help porting over one of his successful iOS apps to Android. I was up for the challenge and it was the first time for me to be exposed to the complicated side of Android development. NodeBeat was written in C++ using openFrameworks. Porting meant that I had to create a native Android UI as well as getting the ‘cross-platform’ common code to work via JNI (some tech details here). Fortunately things worked out well and we released NodeBeat in October of 2011.

While Android was gaining traction, BlackBerry was trying to convince Android developers to make BB10 apps too. Initially I ported over my IRC chat app for a free device and later on I wrote an openFrameworks add-on (ofxQNX) that would allow apps build with that to run on the BlackBerry PlayBook (tablet) and BB10 devices. It also helped me port over NodeBeat to BB10 which got featured on the BlackBerry blog (yay!)!

Sometime in 2012 because of my work on NodeBeat, I was also introduced to Dan Comerchero who needed some help for an Android port of Quiztones, an ear training for EQ app. I don’t remember it being too hard other than the issue where the audio files had to be decompressed WAVs else there would be a playback lag somehow. I guess back then decoding audio from ogg/mp3 wasn’t as smooth.

The biggest challenge was yet to come though… via Twitter in late 2012, Terry Cavanagh was asking around for experienced Android developers and was looking to have his game Super Hexagon ported over to Android. Initially I was contracted to do the Android port but later also worked on the BB10 version and the Ouya game console port.

I remember that I was pretty much done early January 2013 after a few weeks of intense coding and we (me and Terry) were reviewing the game and Terry noticed that something was off on the Nexus 7 (It ran fine on my Nexus One). It seemed like the game was less responsive on that particular device which for an intense game like Super Hexagon matters a lot. Unsure what could be the cause I reached out to Romain Guy on IRC (I wasn’t sure if he was going to respond but I’m glad he did!) I think I asked him because I had seen some other low level graphics related work from him. He put me in touch with Jeff Brown and fortunately he figured out what the cause was. He pointed out that in openFrameworks the physics and render loops were out of sync and recommended us to use the Choreographer to make the game run on Android Jelly Bean devices. The game should still be available from: https://superhexagon.com/

Jeff Brown excited about the game

I’m grateful that the Android team was kind enough to help out, as I’m sure they were all swamped with work for the next Android release.

The Ouya port on a 720p TV…

Running on the Nexus One, BlackBerry PlayBook, Dev Alpha B device (which would become the Z10)

Running on a square screen, Dev Alpha C device (which would become the Q10)

After the different ports for Super Hexagon, I continued writing some apps for BB10. It was basically like the Android early days experience, not a lot of apps where out there so if you were the first, it was easier to become successful. I wrote a utility app SMS Backup for BB10 which allowed you to import/export SMS. It became a success and received the Built for BlackBerry badge that boosted the sales by a lot.

Early 2014 I was contracted by a startup that had technology that would allow you to run Android apps on non-Android platforms such as Tizen. For about 9 months I worked on their internal test tool as well as rewriting their own Android distribution app, the AppMall. It was a great learning experience with many challenges. At some point I had to make the app work on devices like the Pine Neptune which had a tiny screen (320×240). Unfortunately, the company wasn’t going to make it and I had to look for a new job.

The AppMall

The Neptune device had such a tiny screen but I made it work

In 2014 I joined Wanderu and helped creating a successful ground transportation travel app from scratch (for iOS and Android) that is loved by many.

I’m currently (joined late 2022) an Android engineer at Whatnot. We aim to enable anyone to turn their passion into a business & bring people together through commerce.

Whatnot to the moon 🚀!

Inspecting Network Traffic of any Android App

Fri, 07 May 2021 14:05:40 +0000

Like to see what your favorite Android app is doing ? I wrote an article of how you can inspect the network traffic of an app using the Android emulator and mitmproxy check it out here: Inspecting (HTTPS) Network Traffic of any Android App

Creating my own Cycling Trainer App

Sat, 06 Mar 2021 12:35:13 +0000

I’m a big fan of Zwift, it’s a virtual cycling app that allows you to train and compete online in a virtual world.

I started training indoors in November 2018 when the weather was getting colder and I wanted a home setup that I could use if it was raining or snowing outside. Instead of buying an indoor cycling bike or spin bike, I invested in a smart bike trainer (Wahoo Kickr 2018) that I could use with my road bike (Trek Domane SL5). This would allow me to keep using my comfortable bike and it would allow an app to manage the resistance automatically (which isn’t common on a spinbike or a peleton).

The fun part is that Zwift would also allow you to train on courses inspired by real ones. For example, there is Alpe du Zwift which is a virtual copy of Alpe d’Huez in France:

Zwift only allows you to ride the courses they provide, some are made up, others are inspired by real cities/courses.

But… what if I want to ride somewhere that is more familiar to me? I spend some weeks understanding how to control the smart trainer via ANT+ / Bluetooth and started to create my own bike riding software using Unreal Engine 4.

It’s still WIP, doesn’t have pretty rider graphics but it does read the sensor data (Power in Watt), runs physics calculations on it and turns that into speed (similar to what Zwift does). It then applies the slope % to the Wahoo smart bike trainer for the appropriate resistance and also read the Cadence/Heart rate sensor data. During the ride it will record your stats into a fit file that you can upload to services like Strava.

Basically I created a flow (by writing custom UE4 plugins) where I can import an elevation map (DTM), road information and turn it into an UE4 landscape. This means that I don’t have to create the virtual world with too much manual effort and allows me to recreate a map of any place in the world as long as there is enough (elevation) data available. Fortunately for the Boston area, there is very precise data available (1 meter precision).

I took a ‘virtual ride’ at my old neighborhood. With the super steep Lowell St.:

Another experiment was Mount Greylock which is in Western Mass:

Left Unreal Engine 4, Right Strava

NodeBeat featured on the BlackBerry Dev blog

Thu, 02 Aug 2012 18:27:51 +0000

Interested to read how we (me and Seth) ported NodeBeat to the BlackBerry PlayBook platform?

The BlackBerry Developer Blog wrote an article about us, our app and the journey that was involved with it: http://devblog.blackberry.com/2012/07/dev-success-nodebeat/

PlayBook add-on (ofxQNX)

Thu, 26 Apr 2012 23:34:48 +0000

After receiving my BlackBerry PlayBook through the developer offer in March, I started to think of the possibility of porting NodeBeat to the PlayBook platform.

As this is my first tablet (yea I know, as a multitouch enthusiast its quite weird not having an iPad or Android Tablet), I have to say that I am really enjoying the 7 inch form factor. It feels much nicer in the hand compared to most of the larger tablets (10+ inch) which can tire out your wrists quite a bit. Spec wise this device is great (dual-core cpu, 1 GB ram, dedicated graphics chip), therefor it would be nice to test if the PlayBook would be a perfect platform to run NodeBeat on.

Since I am already quite familiar with the NodeBeat code base, this would be a nice new challenge. After taking a brief look at the Native SDK, I figured that this would not be as difficult compared to the Android port. Unlike on Android, there is no need for JNI bindings. Actually, I would say that developing for the PlayBook is very similar to writing applications for Linux (you can use Makefiles if you want), the compiler (qcc) is quite similar to gcc.
RIM includes a modified IDE (based on Eclipse) for app development and it works quite well. All the required tools are tightly integrated and the wizards help you to get through the setup. RIM also did a good job on documenting the Native SDK, including the documents on porting.

Porting

NodeBeat is build on top of the popular openFrameworks (OF) platform, so the first thing I had to do was find out which dependencies I had to build to make it work and write an PlayBook add-on (ofxQNX) to extend the current framework.

At first sight, this seemed to be quite a difficult task due the amount of dependencies OF relies on and the large code base I had to patch, but with help from the openFrameworks community and BlackBerry community, it fortunately did not take long before I had some stuff up and running.

WIP ofxQNX

I won’t bore you guys to death with the details on how to get the dependencies compiled using the NDK as most of them worked straight out of the box (in the readme are some details on what to patch). The ones that were troublesome and caused me a lot of frustration were the ones that use custom build scripts (yes you POCO). I am sure those devs have a valid reason for using it, but it makes things a bit overcomplicated. In the case of POCO, the QNX build (PlayBook target) could only be build on a Linux machine. I ended up installing Ubuntu in VirtualBox to compile FreeImage and Poco.

It was worth the effort though as I managed to get OF run properly on my tablet:

Where to get it?

Details about the project can now be found here: ofxqnx

~~The ofxQNX add-on is available on GitHub, in the developPlayBook branch:~~
~~https://github.com/falcon4ever/openFrameworks/tree/developPlayBook~~

It is licensed under the New BSD license and hopefully will become part of the mainstream branch. For now I will maintain it in my own OF fork as quite a lot of the OF core had to be patched.

Included are ten example projects that explain how to use various features of ofxQNX. All project files contain settings for development on the Simulator (x86) as on the PlayBook/BB10 hardware (ARM).

Is it ready for prime time?

It sure is!

With NodeBeat as our guinea pig, Seth and I have been working pretty hard the past weeks to get NodeBeat up and running just in time for BB 10 Jam. Since NodeBeat uses a lot of different OF features, it was a perfect way to test out ofxQNX and the stability of the add-on.

Below is some footage I shot of an early NodeBeat Beta running on the PlayBook. I love that compared to the Android build, this device is giving us much lower input and audio latencies which really enhance the experience.

We have submitted NodeBeat to the BlackBerry AppWorld so hopefully it will be available soon for your listening pleasure!

NodeBeat beta:

NodeBeat running on the BlackBerry Playbook Tablet

Feedback / Todo

Since ofxQNX is now available to everyone on GitHub, I’d love to hear what you think about using openFrameworks for your PlayBook projects. While a lot of basic OF functionality is already available in ofxQNX, the things that are still lacking are:

ofSoundPlayer, used for controlling the audio levels from the app and play wav/mp3 audio files. For now I recommend using SDL for this purpose.
Camera support. Unfortunately the current PlayBook SDK doesnt allow access to the cameras. As soon as those are supported (prob with the next release) I’ll be porting the cameraExample and openCVExample.
GPS Support with gpsExample

Probably the most important one to focus on is ofSoundPlayer. As there is already an OpenAL implementation (ofOpenALSoundPlayer) in the repository, I am currently investigating if we will be able to (re)use it for ofxQNX.

Anyway, let me know how ofxQNX runs and don’t hesitate to report bugs or submit patches (on GitHub). Enjoy!

[update 05/05/2012]
ofxQNX now also supports the BlackBerry 10 Beta platform!

[update 14/09/2012]
Updated link to the ofxQNX project, now compatible with OF0071

Rewriting and porting fIRC

Wed, 22 Feb 2012 21:55:36 +0000

Late 2008 I created my first Android application (fIRC) after obtaining the T-Mobile G1. The project was an early attempt to master the Java programming language and also a way for me to learn more about the Android mobile platform.

Since the Android Market was still young in 2008, my chat application fIRC became a success. While internet group chat isn’t something new, chatting on your smartphone in real time to other people was. People loved using fIRC because it was easy to join the chatroom for some small talk (By default fIRC connects the chatroom #android-chat on Freenode).

The reviews back in 2009 were pretty favorable, but went downhill after new Android devices were released onto the market. One of the (beginners) mistakes I made when creating fIRC, was that I designed the app specifically for one device, the device I owned: The T-Mobile G1. This means that it was created for devices that had a hardware keyboard, a 480×320 resolution display that was used in landscape mode.

On new devices (let’s say Android 2.x or newer) fIRC didn’t behave that well. UI elements weren’t aligned properly and the bundled resources were only meant to be used on low or medium DPI displays, so on those new devices fIRC looked quite horrible.

As the old chat code turned into a code spaghetti while trying to fix the problems, I decided to rewrite the app from scratch. I did most of the core last summer and some of the UI during fall. The newly rewritten UI of fIRC now scales properly on any phone or tablet device.

New Features

The main new features:

(1) Profile wizard

IRC is not that common these days, one of the complaints of the previous version was that nobody knew how to connect to a different server and how to enter multiple chat rooms (or actually so called “channels”). In order to address this problem, I’ve made a profile wizard, that helps you out by providing some quick settings and a list of commonly used IRC servers (such as DALnet, EFnet and Freenode).

(2) Multi-server

More advanced users requested muti-server supprot because they often like to join multiple irc servers. This new version allows you to do that. Simply create one profile per server, hit the menu button and choose “Connect all profiles” to connect.

(3) Fast channel switching

No need to dive into the menu anymore to switch channels, just use a swipe motion (left/right) to switch between chat rooms.

(4) Fully customizable chat

fIRC now allows you to customize the incoming messages and color them in the way you want. You decide which font, font size, background color and text color you want to user per message type.

(5) File transfer support

A unique feature on Android IRC apps: fIRC supports DCC file transfers (3G and WiFi). Currently only DCC receive is implemented.

(6) Store chat logs

fIRC allows you to store your chat logs on your SD card for future reading.

Porting to the BlackBerry Playbook

Earlier this month Alex Saunders from RIM tweeted:

“Shh…. Android Devs – submit your Android app to BB AppWorld by Feb 13 and get a free Playbook –> tools here: http://bit.ly/w758Ci“.

I wasn’t aware that the BlackBerry Playbook had some kind of emulation layer to run Android apps, so after watching the video below, I decided to give it a shot. I mean, how hard could it possibly be?

Basically you will have to sign up as a vendor on this site: https://appworld.blackberry.com/isvportal/ . RIM will sent you an e-mail (took about a day here) with the request to provide some documents proving you are a company. Fortunately I still had a copy from the Chamber of Commerce that I could use.
While waiting for RIM to approve your account, you should request a code signing key. You will need this key to sign your app later on. After you’re done with that, open up Eclipse and install BlackBerry Plug-in for Android Development Tools.

Since fIRC doesn’t use any of the advanced features of the Android SDK or Android NDK (not supported), the only things I had to do were:

Convert the Android project by adding a “blackberry nature”
Remove the Android references in the app (CTRL-F “Android”)
Resized the application icon to 86×86 pixels
Test the app in the simulator

From the list above, I’d say that testing the app on the simulator was the most time consuming. Blackberry provides a “BlackBerry Tablet Simulator”, but it is actually a virtual drive image with the Playbook software pre-loaded that you can run in VMWare Player.

If you thought the Android emulator was already slow compared to the iOS simulator, think again. Booting up the Playbook takes quite a bit (it feels like booting a desktop OS), and the interaction with the mouse feels a bit sluggish. After the Tablet OS is fully booted, its time to put it in development mode. This allows you to connect to the simulator from Eclipse. Recompiling the Android project and installing it into the simulator feels almost the same as using the Android emulator. Just hit the play button.

Unfortunately the Playbook simulator isn’t all that great for testing your converted BlackBerry apps. In my case the screen flickering was really bad and fonts were blurred (Native BB apps were alright though). Reading the BlackBerry Dev forums, it seem to be an issue with the simulator, so hopefully it looks good on the real device.

The only real trouble I had was signing the app itself to prepare it for the BlackBerry App World. I was following the steps from the Youtube video above. However, BB changed the names of the dropdown menus in Eclipse. Fortunately the documents describe how to do this properly: Sign your app.

Final thoughts

For me porting fIRC to the BlackBerry platform seemed to be a quite painless experience. However, I would advice you to test apps thoroughly on the Android Emulator and when you’re ready for release, test it out on a real Playbook instead of the simulator. That would probably give you a much smoother workflow.

For me that will hopefully soon be possible. Last week I received an e-mail from RIM that my application was accepted to the BlackBerry App World and that I would receive a free Playbook soon :).

The latest version of fIRC for your chatting pleasure is now available from the Android market (Android devices 2.1 and higher) and the BlackBerry App World (BlackBerry Playbook)!

Epic Windmill

Mon, 06 Feb 2012 15:55:41 +0000

It’s official now, I’m proud owner of my new startup company Epic Windmill.

As this blog served as my online portfolio of research work I contributed to, Epic Windmill will be serving as a place for my creative and casual software projects. For the time being, it will focus on the development of applications for handheld devices such as smartphones and tablets (Android and iOS).

Currently two apps are already available from the Android Market, NodeBeat (a creative audio sequencer) and fIRC (a free chat app). We’re preparing a release for NodeBeat so be sure to follow @NodeBeat for more details. A new fIRC will be released later this month.

– Laurence

p.s. Don’t forget to follow us on Twitter @EpicWindmill and like us on Facebook!

NodeBeat, openFrameworks and Android

Sat, 19 Nov 2011 20:52:49 +0000

Last month we (Seth Sandler and yours truly) released the Android port of the popular iPhone/iPad music application NodeBeat.

NodeBeat was created by Seth Sandler and Justin Windle earlier this year and released in April for the iOS platform. It is an experimental node-based audio sequencer and generative music application. By combining music and playful exploration, NodeBeat allows anyone to create an exciting variety of rhythmic sequences and ambient melodies in a fun and intuitive fashion.

How Does it Work?

Octaves and Notes make up the two types of nodes. Octaves pulse and send messages to Notes within proximity. Each Octave is assigned a random octave and each Note, a random note; therefore, a Note will play in several octaves depending on the Octave it’s connected to. Pause nodes to create your own beats or let them roam free to have them generate their own.

Cross platform development

Because NodeBeat was developed using the C++ based open source framework openFrameworks, I did not expected a lot of trouble getting the core to work on Android. However, since the Android port of openFarmworks is still pretty new (we’re using the development branch) and officially only supported on the Mac and Linux platforms, I decided to put some effort into making it work on Windows as well. I’m a Windows user and developer, so if I can avoid dualbooting, I will :P.

Native Development Kit

As soon as you want to use C or C++ in your Android projects, you will have to install the Native Development Kit (NDK). It basically allows you to compile your code into a library which you can access using JNI calls. While in general it is recommend to code using the SDK in Java for your Android projects (the Dalvik VM with JIT show really good performance), lazy coders (like me :P) are always trying to find ways to reuse existing code. Instead of having a native codebase for iOS (in Objective-C) and Android (in Java), it is nicer to have a shared core in C++ with a thin layer (Obj-C or Java) to interface with it. Sure the NDK might sound intimidating at first sight and Google doesnt recommend it unless you know what you’re doing, but honestly I don’t think its rocket science either. After downloading the NDK, you will need to setup a unix like environment such as MinGW or Cygwin.

For my previous projects I already had MinGW installed (you could use Cygwin, but in general I don’t like their approach). I did a fresh checkout from Github and started to mess around with the Makefiles to see if it would compile.

It turned out that all I had to do is replacing a few IF statements (the ones that are checking the build platform) and make them point to the NDK location on my Windows computer. I’ve created a tutorial that explains the steps if you want to try it out yourself. However, if you want to use openFrameworks for your own Android applications, I would highly recommend just using my openFrameworks fork instead (until they accept my pull request). It includes all the patches from the tutorial and it should be compatibly with the latest NDK. The tutorial actually also explains how to run one of the examples so be sure to check that out.

Porting the GUI

For the UI I wanted to stay as close to the iOS version as possible. As I don’t own an iOS device, Seth gave me some screenshots of NodeBeat running on iOS which I used as a reference.

Since the iOS and Android framework are quite distinct, there are cases where I had to do an alternative implementation. For example, on Android most devices have the following buttons:

Back
Menu
Home
Search

The iOS devices only have one button which brings you back to the home screen. In the original implementation of NodeBeat on iOS, there is a shortcut on the canvas that popup a menubar allowing you to access different option menus. On Android however we can use the options menu which allows us to control the flow of the application.

So instead of writing the menu code in C++, I only had to create a XML file for the option menu. It looks like this:

<?xml version="1.0" encoding="utf-8">
<menu xmlns:android="http://schemas.android.com/apk/res/android">
<item android:icon="@drawable/node_icon" android:id="@+id/node" android:title="Node"/>
<item android:icon="@drawable/audio_icon" android:id="@+id/audio" android:title="Audio"/>
...
</menu>

Basically you define a unique id (this allows you to reference it in the Java source code), tell it which icon to use and give it a title. You need to do this for every entry, but apart from that, the Android framework will handle how to display it (depending on orientation and the number of menu items) After the user touches the menu button on Android, it will popup the option menu:

Other menu elements such as the popup menu for the Audio, Rhythm and Settings menu required a different approach. I could’ve switched Views on Android, but in my opinion that would be a bad UI design decision. The problem with this is that the user would be taken away from the NodeBeat activty. Instead I much more prefer to use the context menu that the Android framework provides. This menu popups up and is placed over your current Activity. While it is on the foreground, the activity in the background is still visible and continues running. Another benefit of this approach is that the user will get immediate feedback when adjusting the audio settings. As like the menu, this UI layout is created entirely in XML.

Example: Recording dialog

In some cases the context menu might be a bit of overkill if you want to let the user decide on a question. For example, in the example below we want to inform the user how to record his NodeBeat creation. All we need is a simple dialog that either lets the user confirm the action or decline it. In such cases Android provides dialogs which can be build with an AlertDialog.Builder

AlertDialog.Builder builder = new AlertDialog.Builder(this);
builder.setMessage("You can stop recording by pressing the record button in the menu again")
       .setCancelable(false)
       .setPositiveButton("Yes", new DialogInterface.OnClickListener() {
           public void onClick(DialogInterface dialog, int id) {
                // Start recording
           }
       })
       .setNegativeButton("No", new DialogInterface.OnClickListener() {
           public void onClick(DialogInterface dialog, int id) {
                dialog.cancel(); // Cancel dialog
           }
       });
AlertDialog alert = builder.create();
alert.setTitle("Start recording");

Using JNI Callbacks

After we’ve ported the UI, we still need to pass on our settings to the core application. Fortunately we can use JNI callbacks to get and set NodeBeat’s properties. It is good to know that you should minimize the number of JNI callbacks for performance reasons (so don’t go mental and call tons of JNI methods each time you render a frame).

Let say we would like to pass a value from one of the sliders from the Audio context menu to our NodeBeat core. In such case we first create a new static method in our Java source file:

OFActivity.java

public static native void sliderChanged(float v);

It is important to use the static and native keyword when you define your method. This is all we need to do in our java code and this method can be used anywhere in our java class.

Now the tricky part is how to implement the function on the C++ side of your application. It’s not exactly complex, but you will have to pay attention to a few details. Three things are important here:

The namespace (in Java)
The class name
The method name

If we assume we’re implementing this callback in one of the openFrameworks examples, this means:

The namespace (in Java): cc.openframeworks.androidEmptyExample
The class name: OFActivity
The method name: sliderChanged

main.cpp

Here is the code you implement in C++

#include <jni.h>
extern "C"{
void Java_cc_openframeworks_androidEmptyExample_OFActivity_sliderChanged (JNIEnv*  env, jclass  thiz, jfloat value){
// Do smth here
}</jni.h>

As ugly as this method looks like, take a brief look at how it is constructed. It starts with the return type (which is void just like how we specified it in Java). Next it starts with Java_ and is succeeded by the namespace, classname , method name. All dots from the namespace are replaced by underscores and between each element we place an underscore as well.

In the arguments list, the “JNIEnv* env, jclass thiz” part is mandatory (so if you have a something like this: public static native void methodname(), it would be void methodname(JNIEnv* env, jclass thiz)). For our method we want to pass a float as argument. For some reasons you can’t just pass a float, you will need to use JNI mapping types. The float becomes a jfloat.
Note: for booleans you need to compare the value to JNI_TRUE or JNI_FALSE and not to true or false.

Honeycomb Tablets

Unlike the iOS devices from Apple, Android devices run at so many different configurations and API levels, it can be a bit tricky to support all of them. For NodeBeat we decided to create two versions: A phone version and tablet version. We basically distinguish between phones who are running Eclair or Gingerbread (2.2+) and tablets running Honeycomb (3.0). On the Android Market it seems like we only provide one version, but depending on what device you’re using to download the app, it will give you a certain version.

In order to maximize the use of the display, I’m using this line in the AndroidManifest.xml file:

android:theme="@android:style/Theme.NoTitleBar.Fullscreen"

On phones it sets the application to fullscreen. While this works for anything running Android 2.2+, it is a problem on Honeycomb devices. Honeycomb tablets don’t have any physical buttons and when the application is running in fullscreen mode, there is no shortcut to the menu button. This shortcut is normally placed in the top right corner.

Honeycomb’s menu in the top right corner

This means that for the tablet version, we run the application in regular display mode.
Other than that, NodeBeat provides a rich user experience on tablet devices such as the XOOM or Asus transfomer.

Earbleeding masterpiece created by Sharath Patali (professional coder, horrific musician)
Earbleeding masterpiece created by Sharath Patali (professional coder, horrific musician)

Android Market

Publishing the app to the Android market is no hassle at all (we were just a bit unfortunate and had our app pulled down by accident). There is no annual fee (just a one time 25$) and apps are approved instantly. The dev guide provides a comprehensive overview on how to build your project in release mode and how to sign it.

Go get it!

NodeBeat is available on the Android market:
https://market.android.com/details?id=com.AffinityBlue.NodeBeat

Try it out! It’s just a dollar 🙂

Website: http://nodebeat.com/
Twitter: http://twitter.com/NodeBeat

An introduction to emulation

Mon, 28 Mar 2011 22:35:32 +0000

In the early 90s I primarly used a Commodore 64 (C64) for gaming. As Santa never gave us a Nintendo, I used the C64 until the late 90s. At that point I was introduced to the concept of emulators. Emulators are computer programs that allow you to run software that wasn’t designed for your computer platform. For example, with emulators you could run games from the NES or Sega Master System on PCs. Nowadays computers have enough resources to run games from modern consoles such as the Nintendo Gamecube/Wii and Playstation 2.

Of course, there is always the debate whether it is legal to build an emulator. Console manufacturers will always forbid any type of emulation of their systems. Although its a rather grey area as most documents on the chip design are publicly available (e.g. MIPS / ARM instruction sets).

Space Invaders

Since I got excited about emulation in general, I started to do some research on how to build one myself. As I didn’t want to spend days or even weeks to get some results, I looked for a simple system. A friend introduced me to a small system called the Chip 8.
The Chip 8 itself didn’t existed as a hardware system, but was implemented in software. Due to the small instruction set (~30 opcodes), it was a very good candidate to work on as a first emulation project. The instruction set defines the functionality of the CPU. It for example contains instructions that allow the CPU to load or store data, but also perform mathematical tasks such as multiplications.

Having worked on a couple of emulation projects (and emulator plugins), I started to write a guide that will hopefully help out aspiring emulator authors and inspire emulation enthusiast. The guide explains how emulators work and provides a detailed overview on how to write a Chip 8 interpreter from scratch.

Writing a Chip 8 emulator shouldn’t be too time consuming. On average people tend to finish such project in one or two days. It’s a fun project to test out your programming skills and of course quite educational (and it isn’t just for CS majors ;))

If you’re ready to take up the challenge, click the link below! Have fun!

How to write an emulator (CHIP-8 interpreter)

Brix: a Breakout clone