A lot of working, not a lot of breaks

I try to do everything; it is a good problem to have. However it is extremely hard to learn about every aspect of a project or problem, and exhausting. A few of the problems that I have been working on:

Automated REST API Documentation From LiON files (Lithium Object Notation):

I am using RAML (raml.org) as a POC for automated REST API docs. I will provide a tutorial to standing up an editor server to take advantage of their intuitive UI. This was done on Amazon EC2 but was done manually (not using chef or other deployment tools). Take a look at their website, demo and libraries:

• http://raml.org/
• https://github.com/brianmc/raml-store
• https://github.com/mulesoft/api-designer
• https://github.com/mulesoft/api-designer/issues/95
• https://github.com/raml-org/raml-java-parser
• https://repository.mulesoft.org/nexus/content/repositories/public/org/raml/raml-parser/0.9-SNAPSHOT/

Automated Bamboo Test Monitor and Reporting

At Lithium, we use Bamboo to run and monitor our automated tests. However there are often interesting views into the build results that the bamboo website does not offer. I decided to take all of the history from bamboo and put it in a local Elasticsearch instance. This allows us to split up the loops by team, and display "competitive" calculations such as:

• Team/Loop/Build grade
• Team/Loop/Build strict grade
• Team/Loop test writing velocity
• Team/Loop total green time
• Team/Loop all green time
• Team/Loop grades
• Team/Loop durations

What does it matter to you? I am currently developing and hoping to open source a library called "panda". Can you guess what it does? It is a bamboo REST API client; get it! Panda's consume bamboo! Anyway, I am leveraging fasterxml's Json object mapper and annotations to serialize and deserialize everything. I hope to announce that soon, but there are a few other things that need to get done before then.

Picture you ask?! Unfortunately, not yet. I am still in the process of building out the data model and grading framework. It turns out that elasticsearch has the nice feature of automatically indexing strings making "keys" impossible to use until the mapping is fixed. Am I planning to just use a curl command to fix that ... ?

• https://developer.atlassian.com/display/BAMBOODEV/Bamboo+REST+Resources

ElasticsearchMyAdmin

Is is funny how so many new technologies are trying to get away from SQL but so many people start building out wrappers that allow them to have that familiar SQL feel. Guilty as charged.

Lithium has been working to provide StarQL, a parser which provides a query language that has been specially designed to query against noun-centric REST API data models. We are currently using the parser against a Lucene search backing but are hoping to convert our search index to Elasticsearch for the "infinite" scalability.

However, this tool will allow is to build and edit our object models using a GUI interface while updating how elasticsearch indexes (or does not index) each field. I am sure that we will need to build out some type of migrations between mapping definitions when strings are changed from analysed to not-analyzed, but we will climb that mountain when we get there.

Starting To Build A SOA Suite... And I am scraping ProjectEuler.net for stats

What is this?

Lithium Technologies has been developing StarQL as an open source query language parser designed for REST APIs. In addition to the work I have been doing on the parser, I am slowly building up libraries to map that query language to Elasticsearch queries. This is currently done through JSON object definitions and a dropwizard rest api. The api provides dynamic access to CRUD functionality through traditional calls (http verbs) while providing advanced search functionality with StarQL queries.

Current state

These are more notes to myself to keep track of progress.

• The scraper is working and is tested
• Does not retry when getting unexpected results
• Does not fail gracefully
• Api Service
• Requires json object definitions even if you just want to do quick data dump
• Cannot create object definitions on the fly (requires restart)
• Cannot specify multiple indexes in ES. Only a single service datastore for the time being.
• Mapping for optimized search not implemented
• Mapping StarQL queries to elasticsearch queries works with minimal functionality.

What is next?

You can checkout StarQL here https://github.com/lithiumtech/starql

I plan to opensource the ProjectEulerScraper and StarQL Api Service/Client soon.

My Useful Maven Dependencies... Well, useful for me...

General

Guava

<dependency>

    <groupId>com.google.guava</groupId>

    <artifactId>guava</artifactId>

    <version>15.0</version>

</dependency>

JodaTime

<dependency>

    <groupId>joda-time</groupId>

    <artifactId>joda-time</artifactId>

    <version>2.3</version>

</dependency>

Commons Lang

<dependency>

    <groupId>commons-lang</groupId>

    <artifactId>commons-lang</artifactId>

    <version>2.6</version>

</dependency>

HttpClient

<dependency>
<groupId>commons-httpclient</groupId>
<artifactId>commons-httpclient</artifactId>
<version>3.1</version>
</dependency>

Reflection

<dependency>

  <groupId>org.reflections</groupId>

  <artifactId>reflections</artifactId>

  <version>0.9.9-RC1</version>

</dependency>

---

Servers

Dropwizard

<dependency>

    <groupId>com.yammer.dropwizard</groupId>

    <artifactId>dropwizard-core</artifactId>

    <version>0.6.2</version>

</dependency>

---

Datastores

Elasticsearch

<dependency>

    <groupId>org.elasticsearch</groupId>

    <artifactId>elasticsearch</artifactId>

    <version>0.90.3</version>

</dependency>

Jest

<dependency>

    <groupId>io.searchbox</groupId>

    <artifactId>jest</artifactId>

    <version>0.0.4</version>

</dependency>

MongoDB

<dependency>
<groupId>org.mongodb</groupId>
<artifactId>mongo-java-driver</artifactId>
<version>2.10.1</version>
</dependency>

---

Parsing

Jackson

<dependency>

     <groupId>com.fasterxml.jackson.core</groupId>

    <artifactId>jackson-databind</artifactId>

    <version>2.2.3</version>

</dependency>

StarQL

<dependency>

    <groupId>com.lithium</groupId>

    <artifactId>dev-ldn-starql</artifactId>

    <version>1.6</version>

</dependency>

org.json

<dependency>

    <groupId>org.json</groupId>

    <artifactId>json</artifactId>

    <version>20090211</version>

</dependency>

Jsoup

<dependency>

    <groupId>org.jsoup</groupId>

    <artifactId>jsoup</artifactId>

    <version>1.7.3</version>

</dependency>

Jparsec

<dependency>

    <groupId>jparsec</groupId>

    <artifactId>jparsec</artifactId>

    <version>2.0.1</version>

</dependency>

Starting up again

I will start posting on here again soon. This will be a place to keep my thoughts organized so I don't have to keep everything in my head. I will be writing about

• REST API Best Practices
• R Programming and custom R Packages
• Felxible artificial intelligence competition framework.
• Interesting computer science problems or bugs I have worked on.
• Others?

Stay tuned!