Monitoring Using Mobile Devices – Rants

Just enough monitoring to ensure the health of the product. It shouldn’t be a replacement for SNMP or WMI. Requirements of such a system are

  • Real time monitoring of critical subsystems.
  • Simple and few number of interfaces for effective monitoring. (not SNMP or native monitoring and management agent)
  • Notify the user using device notification, SMS or email.
  • Extensible API for plugging in future products.
  • Interfaces for registering other products.

Architecture considerations

Embeddable monitoring agents where each products ships with its own agent. One of the advantages with this approach is not having to maintain any sort of dependencies with the monitoring agent itself. Users can register their devices with the monitoring agents. There are a few disadvantages with this model.

  • Not all organisations allow personal devices to be connected to the office network
  • Monitoring will be limited to the time the device is connected to the network which will make it not very useful.

Ideally we should be able to deliver notifications to the device directly through cell network. Since Apple allows only their Apple Push Notifications Service to be delivered messages like this and they expect each applications to register with the APN, we may have to have a central notification server which can aggregate messages from our customer premises and then forward them to Apple Push notification service. An aggregation server will have to perform necessary authentication and authorisation to ensure that the users requesting for the notification are indeed allowed to receive messages. Organization Application monitoring app on the device can perform these operations before registering with APN.

Custom User Table with Laravel 5.2

I got introduced to Laravel a few months back while searching for an MVC framework in PHP to build web applications. After playing around a bit with CodeIgniter and Laravel, I decided to proceed with Laravel.

I was blown away by the simplicity with which I can create my data model and seed the tables for initial development. It was quick and easy to pick and get going. It started getting a bit complex as I started using my own user class and tried to combine it with the Authentication model that comes with Laravel.

Given below is a quick summary of changes required if you need to use your own user table with the Authentication framework.

    • Follow laravel instructions to scaffold all routes and views needed for authentication
    • Update your model (default is User.php) to indicate your primary key.  Since I don’t use the default id, it looks something like this in my case

    • You may also want to add the following if password columns is also renamed.
    • Update config/app.php to tell Laravel, how users are retrieved. In my case, I use my_user_table instead of default User table.

    • Its bette that you leave the app/Http/routes.php as it is until you build rest of your business logic.
    • Add the following in app/Http/Controllers/Auth/AuthController.php. This tells the Eloquent model that we need to use username instead of email as the unique identifier for login

  • Update resources/views/auth/login.blade.php so that a variable called username is requested instead of email.

One caveat is about the error logging. Laravel does not have enough log messages to track down problems. Mostly exceptions thrown in the lower level are caught and handled internally which defeats the purpose of exceptions completely. I almost spent two days to figure out the variables needed as the login page kept coming back without any errors. It would have been much easier if there had been an error message before the exceptions were thrown.

Elastic Map Reduce – A Practical Approach

Amazon just reminded me that my AWS free tier is getting over tomorrow.  I’ve been wanting to write about my EMR experiments for some time.  I worked on this a couple of months back when I got a chance to experiment with Hadoop.  We used twitter feeds at that time.   My objective was to run the same with a large log file from one of our products.  I’m going to explain the way EMR can be used in a very basic way by using data stored in S3 and scheduling EMR job with a bunch of scripts.

As usual, I’m going to build the whole experiment over a number of steps.  I believe that it is easier to validate your approach in smaller steps like programming.  It is always easier to test your program as you build it instead of trying to see how the program works after a few hundred thousands lines are written.

Step 1: Upload your data and scripts

pigbucketI’m going to use Amazon S3 as the storage for this example.  There could be other methods.  I think S3 is a good option for up to a few gigabytes of data.  As you notice, the pigdata
pinginput bucket pigdatbucket  has all the input and output data folders for this example.

The objective of this exercise is to do a sentiment analysis on a number of tweets from various states in USA.  The result will be placed in the folder output once the EMR job is completed.

Step 2: Crate  EMR cluster

In this step, we create an EMR cluster.  To start with, I leave logging on and use the S3 folder Logs as the place holder for log files.   I always find logging helpful to troubleshoot teething problems.  I disabled Termination Protection as I couldn’t sufficiently debug script issues when I enable this feature as the cluster terminates automatically.

Amazon provides hadoop 1.0.3 or 2.2.0 and pig 0.11.1.1 (as of this writing). This EMR cluster will be launched in one of your EC2 instances or a VPC.  Select the appropriate instance based on your subscription level.

As this example needs only basic Hadoop configuration, this was selected for the Bootstrap Actions. The core of the setup is in the next step where you select the Pig script that you uploaded as the starting Steps. 

emrscript

 

Notice the S3 locations in the above image.  Select the files from the appropriate S3 folders.

You will be able to monitor the running cluster from your Cluster List once  the cluster is created.  clusterlistSelect one of the clusters to view the status and other configuration details.

This example just uses a basic pig script which I modified for pig 0.11.1.1 that Amazon provides.  You may have a need to call an external program from your pig script to work on the data.  Amazon provides a way to upload additional jar for this purpose.

Preparation.

I would recommend testing your pig script locally on a test data before uploading to EMR.  EMR takes a while to get started and produce the output.  The cycle repeats if there are any errors.  I used Hortonworks Hadoop VM for testing my data and scripts. Hortonworks provides the entire Hadoop stack as a preconfigured sandbox which is very easy to use.  This sandbox also includes Apache Ambari for complete system monitoring.  They have a number of easy to do tutorials for anyone to get started quickly on Hadoop, Pig and Hive.

The initial data and scripts for this example came from Manaranjan.

 

Experimenting with Oracle Virtualbox

I have been using VMWare Fusion on my MBP for a while.  I noticed significant performance issues after upgrading to Mavericks.  That is when I decided to try out Oracle Virtualbox.  More importantly some of the devops I was trying such as Vagrant and Docker did have readily available VMs for Virtualbox.  I never bothered to checkout Virtualbox in the past as I owned licenses for VMWare Fusion and VMWare WorkStation.  Staying with VMWare was more productive as I can move around VMs between my development environments and Office work environment.

Storage

The first step was to getting all my existing VMs running on VirtualBox.  I must say that running my SLES and Ubuntu VMs were easier than I thought. All that I need to do was create a new instance and use the same vmdk image from VMWare.    By default VirtualBox will use a SATA/SCSI interface for the disk image.  It worked well for Unix/Linux virtual machines, but for Windows, I had to forcefully use IDE interface.  Do the following for Windows (I tested with 7.x and 8.1) images.

  • Once the VM is created, goto settings and Storage
  • Delete the SCSI instance associated with your vmdk file
  • Add an IDE interface and choose the same vmdk file.

Networking

The next configuration required is with respect to Networking. I normally use a NAT’d environment with specific CIDR for all my development VMs. I can access this private network from my host on VMware WorkStation or Fusion.  It appears that only way to access services running on Virtualbox image on a private interface is through port forwarding.  Even to SSH to to guest OS, you need to forward a host port to 22 on the guest.  Thankfully the network configuration dialog in the VM settings provides an option to do that.  There is an experimental NetWork Address Translation Service in VirtualBox.  I haven’t been able to get that working on my OSX yet.

Shared Folder

Shared folder concepts are a little convoluted on VirtualBox. Apparently they disable the ability to create symbolic links in a shared folder due to some bizarre security reasons.  You need to enable them manually for each shared folders in each VMs.  More importantly, you need to restart the VirtualBox application after enabling them.  Given below is the syntax for enabling the creation of symbolic links on a given volume.

The SHARE_NAME at the end of the parameter should be a full path to the shared folder on your host.

Headless Mode

One of the features I liked in VirtualBox is the headless mode.  You can run a vm in the background without any UI elements.  This saves some memory on your host and typically you can run any linux instances in runlevel 3.  Push shift key while clicking on the Start button or use VBoxManage command line tool to start a VM in a headless mode.

Overall I find the performance of VirtualBox better than Fusion for my workload.  I’m also liking the command line tools and programmability via its rich set of APIs.  Tune in for more of my VirtualBox experiments.

Linux Advanced Routing: Setting up a Mixed Public-Private Network

Recently I had a unique need to have a mix of public and private network on a particular server for some testing.  A number of services were already configured for the public interface. I had to test a particular feature using a NAT environment and the easiest I could think of was to configure the same server with a NAT ifc in the VMWare environment and configure that feature to use this private interface.  Setting up the proper routes where I can reach the server through the public interface or through the router’s port forwarding via the NAT interface was a challenge in this case. 

Network-ppMy networking requirement is something like this.  As the diagram suggests, 164.99.89.77 is the public interface (eth1) and 172.17.2.80 (eth0) is the private interface.  vmnet5 provides the NAT environment with the network 17217.2.0.  My requirement was to reach the guest via eth0 or eth1 from the 164.99 network.  The host (164.99.89.74) also provides port forwarding so that I can connect to the gust via the private interface. 

I realized that I need to make sure that all answers to traffic coming in on a particular interface get answered from that interface. 

After a little research on Linux advanced routing, I stumbled upon this page.

I designed my routing table based on the recommendations from there.  I’m listing the steps I followed for future reference.

  1. Disable reverse-path filtering for both interfaces.  When source and destination traffic to the same IP using different interface occurs, the Linux kernel drop the traffic as potentially spoofed.  This is called reverse-path filtering. 
  2. Create two additional routing tables, say T1 and T2 in /etc/iproute2/rt_tables.   This file will look something like this

    image

  3. Then populate these tables as given below

    ip route add 164.99.0.0 dev eth1 src 164.99.89.77 tabel T1
    ip route add default via 164.99.89.254 table T1
    ip route add 172.17.2.0 dev eth0 src 172.17.2.80 table T2
    ip route add default via 172.17.2.2 table T2

    164.99.0.0 => public network
    164.99.89.77 => IP address of the public interface
    164.99.89.254 => Gateway address for the public network
    172.17.2.0 => Private network
    172.17.2.80 => IP address of the private interface
    172.17.2.2 => Gateway address for the public network

  4. Set up the main routing table.

    ip route add 164.99.0.0 dev eth1 src 164.99.89.77
    ip route add 172.17.2.0 dev eth0 src 172.17.2.80

  5. Then a preferred default route

    ip route add default via 172.17.2.2

  6. Next set up the routing rules

    ip rule add from 164.99.89.77 table T1
    ip rule add from 172.17.2.80 table T2

Above rules will make sure all answers to traffic coming in on a particular interface get answered from that interface 

My routing table looks something like this with the above changes

image

There are a few more desirable routing additions mentioned here

With these changes, I can connect to the server via the public interface or via the private interface with the port forwarding in the router.

image

image

Introducing IWM at NMAMIT

NMAM Institute of Technology, Nitte held an International Level Conference on “Computer Architecture, Networking and Applications” (IC-CANA 2011) at Mangalore on 7th and 8th January, in association with Penn State University, Harrisburg USA. The conference was co-sponsored by Nitte University, Nitte Education Trust, ISTE, NewDelhi. CSI Division-V (E&R), VTU Belgaum, TCS, EMC, Veriguide and Robosoft Technologies. I was invited as one of the speakers in the tutorial session. I presented Intelligent Workload Management in the cloud track on the first day. I co-chaired the thesis presentation along with Dr. Swarnalatha on the second day and participated in a panel discussion on “Relevance of Industry Institute Interaction in the Global Education Scenario”.

Technical Writing and Cloud Computing

One of my technical writers asked me how they can leverage cloud computing.  Interesting thought!!!. From a developers perspective, we have been talking about cloud as a development and testing resource.  For eg.  I can easily scale up my resources using the cloud environment if I need to run a scalability test.  In the past, I used to raise a request and wait for the resources to arrive before I can even start my test; and I used to call this time as the planning phase.  How do you leverage cloud for technical writing?

The immediate benefit came to my mind is online collaboration.   Desktop publishing is going to be a thing of past.  Google Docs (I’ve been using google docs over 4 years now) revolutionized online shared editing though there were other players in the market at that time.  Adobe, Zoho, Whiteboard and a number of others came in, but none of them could attract us as much as Google. The latest addition to this list is Microsoft with its office live and SkyDrive.  BTW, I just noticed that Office Live doesn’t use a secure connection.  Online documents and collaborative editing will help me to update documents the way we add notes or review comments in Acrobat, Word and OpenOfffice documents.  It will be faster for developers and testers to update the documents as and when they find some changes required and a team member from the technical writers group can merge the changes easily.

Another area is ePublication.  I would assume that all documents need to be available (or online conversion when I download) in one or more of the epub format as eBooks and smart phones are going to be used more for reading and getting our work done.  Today, I read my Books24x7 using my Blackberry when ever I get time. 

Other areas worth considering are location aware content which also includes online translation, integration of product documents with user supplied content, etc.  A couple of blogs I read recently support these predictions. 

Team Velocity – Right word at the wrong place

Agile development methods are overly hyped; probably after cloud!!!.  I happened to read SKORKS today.  I agree with his observation, ie. measuring velocity does not replace planning and scheduling on an agile project.  Velocity is just a metric to measure team productivity.  I would compare this to kloc or design complexity metrics of the traditional development.  One difference is that velocity should always be measured in the team level and so, a constant team is more important.  I heard people talking about shuffling people in a team.  The team velocity is invalid or useless if  team members change constantly in a team.  Once indication of team getting a good grasp of the project is the constantly increasing velocity in the initial phase of the project.  As Alan Skorkin says, a proper estimation process is a must to achieve good and consistent velocity for a team.  More about this later…

Google Voice: Another Innovation from Google

Google’s acquisition of Grand Central a while ago was seen as a smart move by industry.  They took almost two years to invite fresh accounts to try out their new beta program, . The good thing about Google Voice (for that matter Grand Central) is that it lets users integrate all of their existing phone numbers and voice mailboxes into one account.  I have been using VoIP from Gizmo for more than 18  months.  I use a SIP device (Linksys ATA PAP2T) to connect my analog phone and make and receive calls as if using a regular line.  Gizmo provides the basic instructions to configure a SIP device.  BSNL blocks (I believe most of the ISPs do this) the SIP Port (5060) and fortunately, sipphone supports port 5061 also for SIP connectivity.  I also have a call in number through Gizmo so that, colleagues at US can reach me anytime. As far as mobile is concerned,  Nokia removed the native VoIP stack from N96.  However, fring is a reasonable alternative (except support for call mute, DTMF etc.)

Another pretty good collaboration by Google and Gizmo a week back added SIP support for Google Voice through Gizmo.  I just got my Google Voice and Gizmo integrated for both incoming and outgoing calls.  Google Voice call quality is amazing.  Its very easy to use and integrates well with your google address book.  Now, my next project is to get my phone, google and ovi to sync contacts and appointments so that I have a single list of contacts across all my collaboration services.

Publishing Your Blog Posts on Your Facebook Wall

Collaboration is all about getting information easily and through the tools that you want to use.  So, in this respect, I need complete connectivity.  Since I started using facebook actively, I wanted my blog posts to appear on my wall automatically.  So, I configured wordbook which takes care of this syndication.  It appears to work fine now.