I have moved.

This blog has move to here.

NodeJS/Heroku Clock server for the ESP8266 module

Somewhere I would like to visit..

So from my last post, I got my ESP8266 up and running. The next step is to implement something into the http web server that allows setting time intervals for when the relay is enabled/disabled. The ESP8266 doesn't have a real time clock component so the ESP8266 doesn't actually know real absolute time other than since when it started. Therefore, I need a way to keep track of time. Luckily, it does have access to the internet so I can just poll network time. I decided not to go with the standard method of NTP because the API is rather difficult to find, not to mention the example sketch that polls time/date from NTP servers looks frightening... and lengthy. Getting a network time is easy, you can just google "Vancouver time" and google has an in-line widget that shows you the time of Vancouver. In fact, thousands of websites actually give you time from different cities. However, performing a GET request from the ESP8266 using these sites doesn't seem like the best of ideas since the returned HTML from the HTTP requests are rather large compared to the 96kb of data ram available on the ESP8266. Since I'm using a gateway RESTful server that handles outbound sms text messages, I can just create an auxiliary service that filters time and outputs as simple JSON response which should reduce the size of these web sites (46kb in my case using timeanddate.com) down to less than 300 bytes. That should be much easier for the ESP8266 modules to gobble down.

The NodeJS server requires a valid country and city string as how timeanddate.com is used. (eg. http://www.timeanddate.com/ worldclock/canada/vancouver). This converts to /clock/canada/vancouver on the Heroku app. On parsing, it uses NodeJS' request package to create a GET request to timeanddate website.

app.get('/clock/:country/:city', function (req, res) {
  var options = { method: 'GET'};
  options.url = "http://www.timeanddate.com/worldclock/" + req.params.country + "/" + req.params.city;
  
  request(options, function (error, response, body) {
    if (error) throw new Error(error);

      /* Parse the date and time into a JSON object */
      var r = parseDateAndTime(body);
      
      res.setHeader("Content-type", "application/json");
      res.send(JSON.stringify(r));
  });
});

With the html response of the timeanddate.com webpage, the html is DOM-parsed using NodeJS' cheerio package (as called from above code snippet parseDateAndTime())... without the date yet.

function parseDateAndTime(timeAndDateHTML){
        /* Let cheerio parse the html */
      $ = cheerio.load(timeAndDateHTML);
  
      /* Create response object */
      var r = {};
      
      /* Save the time as a text as-is for response */
      r.time = $('#ct').text();
      
      var timeString = r.time.toLowerCase().trim();
      
      /* Determine if this is am or pm */
      var isPM = timeString.indexOf("pm");
      
      /* remove am/pm string */
      t = timeString.replace("pm", "").replace("am", "").trim();
      
      /* split the time string using ':' */
      var timeSplitted = t.split(":");
      
      /* Get the hour in 24-format, min, and seconds*/
      r.hour = parseInt(timeSplitted[0]);
      if (isPM) r.hour = r.hour + 12;
      r.min = parseInt(timeSplitted[1]);
      r.sec = parseInt(timeSplitted[2]);
      
      return r;
}

On the Arduino side, the code is now relatively simple using WifiClient and ArduinoJson library.

  // Has to be of type const char *
  const char *host = "esp8266-clock.herokuapp.com"; 

  // Use WiFiClient class to create TCP connections
  WiFiClient client;
  const int httpPort = 80;
  if (!client.connect(host, httpPort)) {
    Serial.println("connection failed");
    return;
  }

  // Make the GET request
  client.print("GET /clock/canada/vancouver HTTP/1.1\r\nHost: esp8266-clock.herokuapp.com\r\n");

  delay(10);
  
  String res = "";

  // Read all the lines of the reply from server and print them to Serial
  while(client.available()){
    String line = client.readStringUntil('\r');
    Serial.print(line);
    res += line;
  }

  /* Convert String to char array */
  char json[line.length()];
  line.toCharArray(json, line.length());


  /* Parse the JSON time */
  StaticJsonBuffer<300> jsonBuffer;

  JsonObject& root = jsonBuffer.parseObject(json);

  const char* timeString = root["time"];
  byte hour              = root["hour"];
  byte min               = root["min"];
  byte sec               = root["sec"];

ESP8266: DIY $10 IoT Smart Plug Prototype Part 1

I call it the NeMo... :P

Belkin WeMos are cool, but very expensive. One of the incentives of getting the Belkin WeMo, aside from automation, is to both environmentally and financially save on electricity by enabling and disabling your electronics through "smart" controls like IFTTT ("If This Then That") . However, one plug will set you back $59.99 CAD. That's pretty hefty in order to save you money. You can plug a power bar to really make use of it but that means you have no individual control to each devices. If you want a full home automation around the house, you may need 5+ of these to really get things going. Therefore, I set out to make my own.

My bill of materials came out to be:

• $2.88 - Esp8266 module
• $1.00 - USB wall charger (AC to 5V DC converter)
• $0.75 - Box enclosure
• $2.00 - 5V Relay module
• $0.10 - LDO Regulator (to drop 5V to 3.3V for the ESP8266 module)
• $3.00 - 3-pronged extension cable

Total: ~ $10.00 CAD

For about $10, I now have a WiFi connected Belkin WeMo clone. That's pretty cool.

Previously I used the NRF24L01+ radio chip to attempt a basic IoT device. While it worked, the NRF24L01+ was not the best solution. The NRF24L01+ doesn't communicate with standard home wireless networks. Which means in order to correctly make your client a real "internet" connected device, you would need some sort of a gateway like a Raspberry Pi that's connected to the internet with a NRF24L01+ module connected as well. This adds another layer to the connection to the internet.

There is an alternative, however. Recently a new low-cost module has been gaining a lot of traction on the internet. It's still slightly more expensive than an NRF24L01+ module but in terms of your overall project budget, it's cheaper than the NRF24L01+ radio. The reason for that is because the ESP8266 is a WiFi chip that also has microcontroller in itself. And folks have even ported Arduino environments to be loadable onto this chip. So rather than having to buy a Arduino Nano/Mini ($2) and connect that to the NRF24L01+ ($2), you'd just replace them with ESP8266 ($3).

When NeMo gets plugged in, it uses the WiFi SSID/password saved in the EEPROM to connect to the right wifi, if it fails to connect or there isn't a WiFi setting in the EEPROM, it resets itself into the Access Point mode. (Kind of like Chromecast..) After the connection is set up, it sends a text to me broadcasting its own ip address.

When the server is set up, the ESP8266 sets up a RESTful server that allows me to enable and disable gpio pins.

That's it for now, I'll post the details in another later post and the step by step guide so I can follow it myself to "manufacture" a batch of these things.

Modding Android SystemUI.apk - UIAutomatorViewer Tool

Previously I looked at how to extract the SystemUI package and analyze the layout files to create mods. However, SystemUI is a gigantic package with a lot of branching resource structures. It is not easy figuring out how the packages are structured. With Web applications running on Chrome and Firefox, this is easy. Simply use the DOM Inspector or related tools to click on the element you want to customize, find the id, and do a quick grep search to figure out what source file that element is referenced in. With Android, there isn't a DOM inspector - there is a UIAutomatorViewer. This is a tool intended to help QA Engineers to identify app elements and create automation scripts in frameworks such as Appium. This tool however can also help facilitate the modding process.

The UIAutomatorViewer is a tool found in Google's Android SDK under sdk/tools/uiautomatorviewer. In conjunction with adb, you can obtain an xml screenshot of the current screen with a connected device by pressing the "Device Screenshot" button on the toolbar (the second one from the left). What this does is it takes a dump of the current screen on your device and dump the xml layout structure to you.

As an example, let's try to mod something. My previous mod for my Teclast X98 Air 3G missed something. The brightness slider isn't centered as shown in the screenshot.

My quick guess is the decoding added a "left" in the layout_gravity property. So we'll need to change the layout_gravity from "center|top|left" back to "center|top". Of course you can do a quick grep to look for the "center|top" keyword in the resources subdirectories and look for each occurrence to see if they are it. That's not the best way to do things. To figure out what the id of that slider is, we put the tablet into the condition we want and press the "Device Screenshot" button in the toolbar to get something like the following image:

Note: I had to keep my finger on the slider in order to get this screenshot

Then what follows is very similar to the DOM Inspectors. Simply click on the slider, the selected xml element will be indicated on the right. Keep clicking on the parent views until you get to the parent view that holds the entire slider dialog. In this case, it happens to a be FrameLayout. The resource-id of this layout is indicated on the bottom right. This also shows many other properties of the views you are inspecting.

The resource-id says com.android.systemui:id/brightness_mirror The first bit is just the package name. The id is really @+id/brightness_mirror So we'll simply need to do a grep on the resources directory of the decoded SystemUI package to look for brightness_mirror.

grep -r . -e 'brightness_mirror'

My returned result is:

$ grep -r . -e "brightness_mirror"
./SystemUI/res/values/public.xml:    <public type="drawable" name="brightness_mirror_background" id="0x7f020004" />
./SystemUI/res/values/public.xml:    <public type="id" name="brightness_mirror" id="0x7f0e0115" />
./SystemUI/res/values/ids.xml:    <item type="id" name="brightness_mirror">false</item>
./SystemUI/res/layout/super_status_bar.xml:    <FrameLayout android:layout_gravity="center|left|top" android:id="@id/brightness_mirror" android:paddingLeft="@dimen/notification_side_padding" android:paddingRight="@dimen/notification_side_padding" android:visibility="gone" android:layout_width="@dimen/notification_panel_width" android:layout_height="wrap_content">
./SystemUI/res/layout/super_status_bar.xml:        <FrameLayout android:background="@drawable/brightness_mirror_background" android:layout_width="fill_parent" android:layout_height="fill_parent" android:elevation="2.0dip">
Binary file ./SystemUI/dist/SystemUI.apk matches
Binary file ./SystemUI/build/apk/resources.arsc matches
./SystemUI/original/META-INF/CERT.SF:Name: res/drawable/brightness_mirror_background.xml
./SystemUI/original/META-INF/MANIFEST.MF:Name: res/drawable/brightness_mirror_background.xml
Binary file ./SystemUI.apk matches
Binary file ./new.apk matches

From this result, I see that the line gives us details about where it is referenced (super_staus_bar.xml):

./SystemUI/res/layout/super_status_bar.xml:    <FrameLayout android:layout_gravity="center|left|top" android:id="@id/brightness_mirror" android:paddingLeft="@dimen/notification_side_padding" android:paddingRight="@dimen/notification_side_padding" android:visibility="gone" android:layout_width="@dimen/notification_panel_width" android:layout_height="wrap_content">

Furthermore, it also shows the layout_gravity property that we're interested in with the value "center|left|top", change that, repackage, push back onto the device and you're good to go.

Here are my edits for my lockscreen, I've removed the carrier text as I'm not using 3G. I've removed the icons for camera and phone because they're not used very often. And I've also removed the cell network signal status icon. Just the way I like it.

Modding Android SystemUI.apk

Look at that bloated ugliness of a navigation bar.. And it's not even centered!

I've just upgraded my Teclast X98 Air 3G tablet to Android Lollipop and it is faaaantastic. It's incredible how each new Android release makes older hardware run the software with ever more fluidity. However, with the new upgrade means I've lost the stock navigation bars. These Chinese tablets love to mod the navigation bar to include the screenshot, vol+, vol-, and the menu icon (most likely to support Android TV sticks. They look ugly and are a nuisance when I keep accidentally tapping on it. Now to remove this requires modding the stock Android system app SystemUI.apk. I've done this before but I couldn't quite remember it hence I'm leaving a note to myself on how this is done so I won't have to keep googling next time around.

Note: For Windows users, all of these information should be the same, the shell scripts will not work though, you will have to adapt it for batch scripting.

What you'll need:

• Rooted Android device with developer options enabled
• Apktool.jar - at time of writing latest is apktool_2.0.1.jar
• Computer with adb tools and Java installed

So, first, you'll need a copy of the SystemUI.apk. Do this via:

adb pull /system/priv-app/SystemUI/SystemUI.apk

You'll also need a copy of framework-res.apk. Again:

adb pull /system/framework/framework-res.apk

These two files basically contain the android system's layout and image resources - the "/res" directory in any UI Android activity. Keep a copy of these somewhere so you can always reset to these if your mods mess up.

Next, we'll need to extract these files.

# install framework
java -jar apktool_2.0.1.jar if ./framework-res.apk

# install systemUI
java -jar apktool_2.0.1.jar if ./SystemUI.apk

# decode the SystemUI package
java -jar apktool_2.0.1.jar d ./SystemUI.apk

This gives us an extracted /SystemUI folder with all the resources as if you were building an Android app project. If you're an Android developer, this should be familiar to you, go ahead and get modding. All you have to do is find the layout for the navigation bar and change the xml files. In my case, the appropriate file is in /SystemUI/res/layout-sw600dp-v13/navigation_bar.xml. This path indicates that the resource is used on a device with a minimum screen width of 600 dp and has at least an SDK version of 13. Now open the file and you'll see a text in xml format. This, here, is interesting to me because it seems that the layout definition for the landscape and portrait modes are actually sitting in the same xml file, but the orientation would just simply pick whether to use the first FrameLayout with id "@id/rot0" or the second FrameLayout with id "@id/rot90". Anyway, looking closely at the ids of some of the KeyButtonViews, you can see "@id/screenshot", "@id/volumedown", "@id/back", "@id/home", "@id/recent_apps", "@id/volume_up", "@id_menu", etc. Now we just have to modify these such that their dimensions are so small that you can't see them. Or you could make their visibility property invisible. However, that would not solve the problem because they'd still be clickable even if they're invisible.

My modifications to these xml items are to set the layout_width to "0.0dip" so that each of these entries would have this property:

android:layout_width="0.0dip"

Save the file. Before we go on to the next step, it is noted that the current apktool decodes this SystemUI.apk incorrectly and will align the Lollipop's pulldown menu to the left when you try to apply this patch, even without mucking any of the xml files. I'm unsure if this is an issue with Apktool or the SystemUI.apk file that is used in this tablet. In any case, the additional steps you'll need to modify are:

• in file /SystemUI/res/layout/status_bar_expanded_header.xml, look for android:layout_gravity="center|left|top" and change it to android:layout_gravity="center|top" (should only be one instance ofStatusBarHeaderView that uses this property)
• in file /SystemUI/res/layout/status_bar_expanded.xml, do the same thing. replace all "center|left|top" to "center|top" (4 instances in my build)

Now we need to package this back into the Android package format using the same Apktool. This is simply done by running:

java -jar apktool_2.0.1.jar b SystemUI

In your previously generated /SystemUI/ folder should now be a sub-folder called "dist" (short for distribution I suppose?). In that sub-folder you will find a SystemUI.apk, this is the newly modded file you will try to put back into your device. To do so, run the following commands.

# Temporarily throw the file back into the sdcard root
adb push ./SystemUI/dist/SystemUI.apk /sdcard/SystemUI.apk

# Get superuser access and mount the system directory as rewritable
adb shell "su -c mount -o remount,rw /system"

# Overwrite the new SystemUI.apk file with the current SystemUI.apk
adb shell "su -c cp /sdcard/SystemUI.apk /system/priv-app/SystemUI/SystemUI.apk"

In a moment your android device may complain that SystemUI.apk has stopped working, and it will flash momentarily, locking your device and on wake/return, you'll see the modded navigation bars. You may have to reboot device for pre-KitKat versions of Android in this step. If your device doesn't complain, lock the screen and wake it up again, it should just load the new modded apk. If it still doesn't do it, try restarting, however, I have not come across this issue yet. I do notice that sometimes small changes do not get updated, so what I would do is use the last set of commands to restore the SystemUI.apk to the original version (before you modded, the one I told you to keep a backup somewhere) and then reapply the new patch. It would be wise to create a script that you will run each time you make a modification to facilitate loading the modded files in. Here are my shell scripts:

# decode.sh
rm -r ~/apktool/framework
rm -r SystemUI

# install framework
java -jar apktool_2.0.1.jar if ./framework-res.apk

# install systemUI
java -jar apktool_2.0.1.jar if ./SystemUI.apk

# decode the SystemUI package
java -jar apktool_2.0.1.jar d ./SystemUI.apk

# encodeAndLoad.sh
java -jar apktool_2.0.1.jar b SystemUI

# Temporarily throw the file back into the sdcard root
adb push ./SystemUI/dist/SystemUI.apk /sdcard/SystemUI.apk

# Get superuser access and mount the system directory as rewritable
adb shell "su -c mount -o remount,rw /system"

# Overwrite the new SystemUI.apk file with the current SystemUI.apk
adb shell "su -c cp /sdcard/SystemUI.apk /system/priv-app/SystemUI/SystemUI.apk"

# restore.sh
adb push ~/SystemUI.apk /sdcard/SystemUI.apk
adb shell "su -c mount -o remount,rw /system"
adb shell "su -c cp /sdcard/SystemUI.apk /system/priv-app/SystemUI/SystemUI.apk"

What does the end result look like?

Much cleaner, but the mdpi drawables for the back/home/recent buttons are kind of fuzzy. I should do something about that, too.