Tuesday, December 17, 2013

A lot of working, not a lot of breaks

I try to do everything; it is a good problem to have. However it is extremely hard to learn about every aspect of a project or problem, and exhausting. A few of the problems that I have been working on:

Automated REST API Documentation From LiON files (Lithium Object Notation):

I am using RAML (raml.org) as a POC for automated REST API docs. I will provide a tutorial to standing up an editor server to take advantage of their intuitive UI. This was done on Amazon EC2 but was done manually (not using chef or other deployment tools). Take a look at their website, demo and libraries:

Automated Bamboo Test Monitor and Reporting

At Lithium, we use Bamboo to run and monitor our automated tests. However there are often interesting views into the build results that the bamboo website does not offer. I decided to take all of the history from bamboo and put it in a local Elasticsearch instance. This allows us to split up the loops by team, and display "competitive" calculations such as:
  • Team/Loop/Build grade
  • Team/Loop/Build strict grade
  • Team/Loop test writing velocity
  • Team/Loop total green time
  • Team/Loop all green time
  • Team/Loop grades
  • Team/Loop durations
What does it matter to you? I am currently developing and hoping to open source a library called "panda". Can you guess what it does? It is a bamboo REST API client; get it! Panda's consume bamboo! Anyway, I am leveraging fasterxml's Json object mapper and annotations to serialize and deserialize everything. I hope to announce that soon, but there are a few other things that need to get done before then.

Picture you ask?! Unfortunately, not yet. I am still in the process of building out the data model and grading framework. It turns out that elasticsearch has the nice feature of automatically indexing strings making "keys" impossible to use until the mapping is fixed. Am I planning to just use a curl command to fix that ... ?

ElasticsearchMyAdmin

Is is funny how so many new technologies are trying to get away from SQL but so many people start building out wrappers that allow them to have that familiar SQL feel. Guilty as charged.

Lithium has been working to provide StarQL, a parser which provides a query language that has been specially designed to query against noun-centric REST API data models. We are currently using the parser against a Lucene search backing but are hoping to convert our search index to Elasticsearch for the "infinite" scalability.

However, this tool will allow is to build and edit our object models using a GUI interface while updating how elasticsearch indexes (or does not index) each field. I am sure that we will need to build out some type of migrations between mapping definitions when strings are changed from analysed to not-analyzed, but we will climb that mountain when we get there.


Thursday, November 21, 2013

Starting To Build A SOA Suite... And I am scraping ProjectEuler.net for stats

What is this?

Lithium Technologies has been developing StarQL as an open source query language parser designed for REST APIs. In addition to the work I have been doing on the parser, I am slowly building up libraries to map that query language to Elasticsearch queries. This is currently done through JSON object definitions and a dropwizard rest api. The api provides dynamic access to CRUD functionality through traditional calls (http verbs) while providing advanced search functionality with StarQL queries.



Current state

These are more notes to myself to keep track of progress.

  • The scraper is working and is tested
    • Does not retry when getting unexpected results
    • Does not fail gracefully
  • Api Service
    • Requires json object definitions even if you just want to do quick data dump
    • Cannot create object definitions on the fly (requires restart)
    • Cannot specify multiple indexes in ES. Only a single service datastore for the time being.
    • Mapping for optimized search not implemented
    • Mapping StarQL queries to elasticsearch queries works with minimal functionality.

What is next?

You can checkout StarQL here https://github.com/lithiumtech/starql

I plan to opensource the ProjectEulerScraper and StarQL Api Service/Client soon.

Wednesday, November 20, 2013

My Useful Maven Dependencies... Well, useful for me...

General

Guava

<dependency>
    <groupId>com.google.guava</groupId>
    <artifactId>guava</artifactId>
    <version>15.0</version>
</dependency>

JodaTime

<dependency>
    <groupId>joda-time</groupId>
    <artifactId>joda-time</artifactId>
    <version>2.3</version>
</dependency>

Commons Lang

<dependency>
    <groupId>commons-lang</groupId>
    <artifactId>commons-lang</artifactId>
    <version>2.6</version>
</dependency>

HttpClient

<dependency>
<groupId>commons-httpclient</groupId>
<artifactId>commons-httpclient</artifactId>
<version>3.1</version>
</dependency>

Reflection

<dependency>
  <groupId>org.reflections</groupId>
  <artifactId>reflections</artifactId>
  <version>0.9.9-RC1</version>

</dependency>





Servers

Dropwizard

<dependency>
    <groupId>com.yammer.dropwizard</groupId>
    <artifactId>dropwizard-core</artifactId>
    <version>0.6.2</version>
</dependency>




Datastores

Elasticsearch

<dependency>
    <groupId>org.elasticsearch</groupId>
    <artifactId>elasticsearch</artifactId>
    <version>0.90.3</version>
</dependency>

Jest

<dependency>
    <groupId>io.searchbox</groupId>
    <artifactId>jest</artifactId>
    <version>0.0.4</version>
</dependency>

MongoDB

<dependency>
<groupId>org.mongodb</groupId>
<artifactId>mongo-java-driver</artifactId>
<version>2.10.1</version>
</dependency>





Parsing

Jackson

<dependency>
     <groupId>com.fasterxml.jackson.core</groupId>
    <artifactId>jackson-databind</artifactId>
    <version>2.2.3</version>
</dependency>

StarQL

<dependency>
    <groupId>com.lithium</groupId>
    <artifactId>dev-ldn-starql</artifactId>
    <version>1.6</version>
</dependency>

org.json

<dependency>
    <groupId>org.json</groupId>
    <artifactId>json</artifactId>
    <version>20090211</version>
</dependency>

Jsoup

<dependency>
    <groupId>org.jsoup</groupId>
    <artifactId>jsoup</artifactId>
    <version>1.7.3</version>
</dependency>

Jparsec

<dependency>
    <groupId>jparsec</groupId>
    <artifactId>jparsec</artifactId>
    <version>2.0.1</version>
</dependency>

Thursday, April 18, 2013

Starting up again

I will start posting on here again soon. This will be a place to keep my thoughts organized so I don't have to keep everything in my head. I will be writing about


  • REST API Best Practices
  • R Programming and custom R Packages
  • Felxible artificial intelligence competition framework.
  • Interesting computer science problems or bugs I have worked on.
  • Others?
Stay tuned!

Friday, July 13, 2012

Java Socket Chat Application v0.1

This is a project for a couple of my CS1332 students. I have been working with them explaining how sockets and networking works specifically related to Java sockets. This is a quick proof of concept application that I threw together just to make sure that I was leading them in the right direction.

Right now it is an infinite loop that checks to see if the user has submitted text to the console or if the socket has received a string. All text is logged to an output file on each machine with the correct prefix to each message (">>" for incoming and "<<" for outgoing).



Code can be found below.

Server.java

import java.io.IOException;
import java.net.ServerSocket;
import java.net.Socket;

public class Server {

    public static void main(String[] args) {
        try {
            ServerSocket sSocket = new ServerSocket(3000);
            Socket socket = sSocket.accept();
            System.out.println(\"Client accepted\");
            Chatter c = new Chatter(socket, \"Server.log\");
            c.run();
        } catch (IOException e) {
            // ignore
        }
    }
}

Client.java

import java.io.IOException;
import java.net.ServerSocket;
import java.net.Socket;

public class Server {

    public static void main(String[] args) {
        try {
            ServerSocket sSocket = new ServerSocket(3000);
            Socket socket = sSocket.accept();
            System.out.println(\\\"Client accepted\\\");
            Chatter c = new Chatter(socket, \\\"Server.log\\\");
            c.run();
        } catch (IOException e) {
            // ignore
        }
    }
}

Chatter.java

import java.io.BufferedInputStream;
import java.io.BufferedWriter;
import java.io.File;
import java.io.FileWriter;
import java.io.IOException;
import java.io.ObjectInputStream;
import java.io.ObjectOutputStream;
import java.net.Socket;
import java.text.SimpleDateFormat;
import java.util.Date;
import java.util.Scanner;

public class Chatter {
    private Socket socket;
    private BufferedInputStream bis;
    private ObjectInputStream ois;
    private ObjectOutputStream oos;
    private BufferedInputStream console_bis;
    private Scanner scanner;
    private BufferedWriter bfw;

    public Chatter(Socket socket, String logName) {
        try {
            this.socket = socket;
            this.bis = new BufferedInputStream(socket.getInputStream());
            this.oos = new ObjectOutputStream(socket.getOutputStream());
            this.console_bis = new BufferedInputStream(System.in);
            this.scanner = new Scanner(console_bis);
            this.bfw = new BufferedWriter(new FileWriter(new File(logName), true));
            this.bfw.append(\"\\n\\n\"
                    + new SimpleDateFormat(\"yyyy/MM/dd HH:mm:ss\")
                            .format(new Date()) + \"\\n===================\\n\");
            this.bfw.flush();
            
            this.ois = new ObjectInputStream(bis);
        } catch (IOException e) {
            e.printStackTrace();
        }
    }

    public void run() {
        String temp = \"\";
        while (temp == null || !temp.equals(\"quit\")) {
            temp = readSocketMessage();
            if (temp != null) {
                if (temp.equals(\"quit\")) {
                    System.out.println(\"[[ Other User Logged Off ]]\");
                    close();
                    System.exit(0);
                }
                logMessage(temp, true);
            }
            temp = readConsoleMessage();
            if (temp != null) {
                sendMessage(temp);
                logMessage(temp, false);
            }
        }
        this.close();
    }
    
    private void close() {
        try {
            socket.close();
        } catch (IOException e) {
            // ignore
        }
        try {
            bfw.close();
        } catch (IOException e) {
            // ignore
        }
    }

    private String readSocketMessage() {
        try {
            if (bis.available() > 0) {
                return (String) ois.readObject();
            }
        } catch (IOException e) {
            System.err
                    .println(\"Fatal Socket Exception: could not read message.\");
            System.exit(0);
        } catch (ClassNotFoundException e) {
            System.err.println(\"Socket Exception: peer sent bad input.\");
        }
        return null;
    }

    private String readConsoleMessage() {
        try {
            if (console_bis.available() > 0)
                return scanner.nextLine();
        } catch (IOException e) {
        }
        return null;
    }

    private void sendMessage(String message) {
        try {
            oos.reset();
            oos.writeObject(message);
            oos.flush();
        } catch (IOException e) {
            System.err
                    .println(\"Fatal Socket Exception: could not send message.\");
            System.exit(0);
        }
    }

    private void logMessage(String message, boolean display) {
        try {
            if (display) {
                System.out.println(\">> \" + message);
                bfw.append(\">> \");
            } else
                bfw.append(\"<< \");
            bfw.append(message).append(\"\\n\");
        } catch (IOException e) {
            e.printStackTrace();
        }
    }
}

Sunday, July 8, 2012

Whiteboard Clock


Introduction

I have been working on a calculator for my girlfriend, but unforeseen complexity has caused the project to stretch out over months rather than weeks. In place of finishing the calculator, I came up with a more simple, more useful project (birthday present) which has the potential to be profitable if the design is tweaked to be easier to manufacture. This post contains (1) a video of the finished product (2) followed by background information (3) followed by a gallery detailing each step of the build process and (4) finally the Arduino code behind the functionality.


Lizz's Whiteboard Clock


Background

Each led requires a high signal to turn on. This means that I need a total of 72 separate signals to turn on each led individually. However an Arduino only has 14 digital output. If we consider using pins in combinations there are a few methods of controlling large number of signals using a small number of I/O ports.

  1. Multiplexing: This allows a microprocessor to control 2^n signals with n outputs. These are used heavily when building custom hardware (verilog, VHDL, etc.) but there are not large IC chips available. I would have to string multiple smaller DEMUX chips together to achieve a 7 bit DEMUX.
  2. Charlieplexing: This allows a microprocessor to control n(n-1) signals with n outputs. Charlieplexing relies on the fact that LED are directional i.e. they will only turn on when a voltage is applied to then in the correct direction. This means that we can have an LED hooked up to pin 1 and 2 and another LED hooked to pin 2 and 1 while turning them on or off independently.
Both of these methods allows for a single LED to be turning on at any given time. However, for a clock we need at most 2 or 3 LEDs turned on at once (hours, minutes, seconds). We can accomplish this using persistence of vision. Most video on the internet is only 30 frames per second (updates 30 times a second) because the human eye cannot perceive changes faster than that. This means that if we update the clock 30 times per second while cycling through hours, minutes and seconds, then we can make each LED appear to be on simultaneous to the other two. There are specialized chips which do this (MAX7221) for 7-segment display and LED dot-matrices.


The Build


The Code

LizzClock.h

#ifndef LIZZCLOCK_h
#define LIZZCLOCK_h


typedef struct LED {
  char high;
  char low;
} LED;

typedef struct BUTTON {
  long down;
  char set;
  char PIN;
} BUTTON;


typedef struct TIME {
  char hours;
  char minutes;
  char seconds;
} TIME;

#endif

LizzClock.ino

#include "LizzClock.h"
#include <TimerOne.h>

#define TRUE    1
#define FALSE   0

// Display Constants
#define OFFSET  2

LED lights[72];
LED* curr_led = 0;
char PAUSE = 50;
char BRIGHTNESS = 0;
int UPDATE_COUNT (100/PAUSE);
int ON_DELAY = PAUSE;
int OFF_DELAY = 0;

TIME *curr_time;
long start_time;

BUTTON *minutes_button;
BUTTON *hours_button;

void setup() {
  /*
   * Initalize current time and update interrupt
   */
  start_time = millis();
  curr_time = (TIME*) malloc(sizeof(TIME));
  curr_time->seconds = 0;
  curr_time->minutes = 0;
  curr_time->hours = 0;
  Timer1.initialize(100000);
  Timer1.attachInterrupt(update_time);

  /*
   * Initalize buttons
   */
  minutes_button = (BUTTON*)malloc(sizeof(BUTTON));
  minutes_button->down = 0;
  minutes_button->PIN = 12;
  pinMode(minutes_button->PIN, INPUT);
  hours_button = (BUTTON*)malloc(sizeof(BUTTON));
  hours_button->down = 0;
  hours_button->PIN = 13;
  pinMode(hours_button->PIN, INPUT);

  /*
   * Initalize display
   */
  set_brightness(30);  

  /*
   * MINUTES/SECONDS
   */
  init_LED( 0, 0, 1); // 0  and 1
  init_LED( 2, 0, 2); // 2  and 3 
  init_LED( 4, 1, 2); // 4  and 5
  init_LED( 6, 0, 3); // 6  and 7
  init_LED( 8, 1, 3); // 8  and 9
  init_LED(10, 2, 3); // 10 and 11
  init_LED(12, 0, 4); // 12 and 13
  init_LED(14, 1, 4); // 14 and 15
  init_LED(16, 2, 4); // 16 and 17
  init_LED(18, 3, 4); // 18 and 19
  init_LED(20, 0, 5); // 20 and 21
  init_LED(22, 1, 5); // 22 and 23
  init_LED(24, 2, 5); // 24 and 25
  init_LED(26, 3, 5); // 26 and 27
  init_LED(28, 4, 5); // 28 and 29
  init_LED(30, 0, 6); // 30 and 31
  init_LED(32, 1, 6); // 32 and 33
  init_LED(34, 2, 6); // 34 and 35
  init_LED(36, 3, 6); // 36 and 37
  init_LED(38, 4, 6); // 38 and 39
  init_LED(40, 5, 6); // 40 and 41
  init_LED(42, 0, 7); // 42 and 43
  init_LED(44, 1, 7); // 44 and 45
  init_LED(46, 2, 7); // 46 and 47
  init_LED(48, 3, 7); // 48 and 49
  init_LED(50, 4, 7); // 50 and 51
  init_LED(52, 5, 7); // 52 and 53
  init_LED(54, 6, 7); // 54 and 55
  init_LED(56, 0, 8); // 56 and 57
  init_LED(58, 1, 8); // 58 and 59

  /*
   * HOURS
   */
  init_LED(60, 2, 8); // 12 and 1
  init_LED(62, 3, 8); // 2  and 3
  init_LED(64, 4, 8); // 4  and 5
  init_LED(66, 5, 8); // 6  and 7
  init_LED(68, 6, 8); // 8  and 9
  init_LED(70, 7, 8); // 10 and 11
}

/*\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\
 *      [[  DISPLAY INIT START  ]]
 */

char set_brightness(int new_brightness) {
  if (new_brightness >= 0 && new_brightness <= PAUSE) {
    BRIGHTNESS = new_brightness;
    ON_DELAY = (PAUSE > BRIGHTNESS ? PAUSE - BRIGHTNESS : 0);
    OFF_DELAY = (PAUSE < BRIGHTNESS ? PAUSE : BRIGHTNESS);
    return TRUE;
  }
  return FALSE;
}

void init_LED(char index, char high, char low) {
  lights[index+1].high = high;
  lights[index+1].low = low;
  lights[index].high = low;
  lights[index].low = high;
}
//      [[  DISPLAY INIT END  ]]
//////////////////////////////////////////////////////////


/*\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\
 *      [[  EXECUTION START  ]]
 */
void loop() {
  // update_time called on timed interrupt 10 times a second
  display_time(curr_time);
  check_user_set_time();
}
//      [[  EXECUTION END  ]]
//////////////////////////////////////////////////////////


/*\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\
 *      [[  MAINTAIN DISPLAY START  ]]
 */
LED* get_hour(char hour) {
  if (hour <= 0 || hour >= 12)
    return &lights[60];
  return &lights[hour + 60];
}

void all_off()
{
  for(int i = 2; i < 12; i++) {
    pinMode(i, INPUT);
    digitalWrite(i, LOW);
  }
}

char get_pin(char index) {
  return index + OFFSET;
}

void display_time(TIME *time) {
  lightup(&lights[time->seconds]);
  lightup(&lights[time->minutes]);
  lightup(get_hour(time->hours));
}

void lightup(LED* led) {
  // blank display and pause to control brightness
  all_off();
  delayMicroseconds(OFF_DELAY);
  //delayMicroseconds();

  // turn on display for short time to allow for persistance of vision.
  pinMode(get_pin(led->high), OUTPUT);
  digitalWrite(get_pin(led->high), HIGH);
  pinMode(get_pin(led->low), OUTPUT);
  digitalWrite(get_pin(led->low), LOW);
  curr_led = led;
  delayMicroseconds(ON_DELAY);
}
//      [[  MAINTAIN DISPLAY END  ]]
//////////////////////////////////////////////////////////


/*\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\
 *    [[  MAINTAIN CLOCK START  ]]
 */
void update_time() {
  long temp = (millis() - start_time) / 1000.0;
  //long temp = (millis() - start_time) / 3;
  curr_time->seconds = temp % 60;
  temp /= 60.0;
  curr_time->minutes = temp % 60;
  curr_time->hours = temp / 60.0;
  if (curr_time->hours == 12) {
    curr_time->hours = 0;
    // assume time is updated more than once a second.
    start_time = millis() - (curr_time->seconds * 1000);
    //start_time = millis() - (curr_time->seconds * 3);
  }
}

void check_user_set_time() {
  if (digitalRead(minutes_button->PIN)) {
    // if button down
    if (minutes_button->down) {
      // if currently debouncing
      if (!minutes_button->set && millis() - minutes_button->down > 20) {
        // if debounced long enough
        start_time -= 60000; // seconds * milliseconds
        minutes_button->set = 1;
      }
    } 
    else {
      // start debouncing
      minutes_button->down = millis() ^ 1;
    }
  } 
  else if (minutes_button->down) {
    minutes_button->down = 0;
    minutes_button->set = 0;
  }
  if (digitalRead(hours_button->PIN)) {
    // if button down
    if (hours_button->down) {
      // if currently debouncing
      if (!hours_button->set && millis() - hours_button->down > 20) {
        // if debounced long enough
        start_time -= 3600000; // minutes * seconds * milliseconds
        hours_button->set = 1;
      }
    } 
    else {
      // start debouncing
      hours_button->down = millis() ^ 1;
    }
  } 
  else if (hours_button->down) {
    hours_button->down = 0;
    hours_button->set = 0;
  }
}
//      [[  MAINTAIN CLOCK END  ]]
//////////////////////////////////////////////////////////

Thursday, June 14, 2012

Towel Build and Parts

The Towel is a simple RC delta wing poularized by Brooklyn Aerodrom and MAKE Online. It is a simple build for under $100 that should allow flight in only a few hours work. I stumbled across HobbyPartz.com which has amazing deals on robotics, RC and hobby electronics parts.

The following parts are either on their way for my Towel build or worthy of mention: 


  • 2.4G FlySky 6-Channel TX/RX Combo for $33 [[ Here ]] 
  • 2.4G FlySky 6-Channel Receiver for $9 [[ Here ]] 
  • Brushless Motor (KV 1300) + 18A ESC + 3 x 9g Servos for $25 [[ Here ]] 
I will post as progress is made. However, research will not let that happen anytime soon...