Starting To Build A SOA Suite... And I am scraping ProjectEuler.net for stats

What is this?

Lithium Technologies has been developing StarQL as an open source query language parser designed for REST APIs. In addition to the work I have been doing on the parser, I am slowly building up libraries to map that query language to Elasticsearch queries. This is currently done through JSON object definitions and a dropwizard rest api. The api provides dynamic access to CRUD functionality through traditional calls (http verbs) while providing advanced search functionality with StarQL queries.

Current state

These are more notes to myself to keep track of progress.

• The scraper is working and is tested
• Does not retry when getting unexpected results
• Does not fail gracefully
• Api Service
• Requires json object definitions even if you just want to do quick data dump
• Cannot create object definitions on the fly (requires restart)
• Cannot specify multiple indexes in ES. Only a single service datastore for the time being.
• Mapping for optimized search not implemented
• Mapping StarQL queries to elasticsearch queries works with minimal functionality.

What is next?

You can checkout StarQL here https://github.com/lithiumtech/starql

I plan to opensource the ProjectEulerScraper and StarQL Api Service/Client soon.

My Useful Maven Dependencies... Well, useful for me...

General

Guava

<dependency>

    <groupId>com.google.guava</groupId>

    <artifactId>guava</artifactId>

    <version>15.0</version>

</dependency>

JodaTime

<dependency>

    <groupId>joda-time</groupId>

    <artifactId>joda-time</artifactId>

    <version>2.3</version>

</dependency>

Commons Lang

<dependency>

    <groupId>commons-lang</groupId>

    <artifactId>commons-lang</artifactId>

    <version>2.6</version>

</dependency>

HttpClient

<dependency>
<groupId>commons-httpclient</groupId>
<artifactId>commons-httpclient</artifactId>
<version>3.1</version>
</dependency>

Reflection

<dependency>

  <groupId>org.reflections</groupId>

  <artifactId>reflections</artifactId>

  <version>0.9.9-RC1</version>

</dependency>

---

Servers

Dropwizard

<dependency>

    <groupId>com.yammer.dropwizard</groupId>

    <artifactId>dropwizard-core</artifactId>

    <version>0.6.2</version>

</dependency>

---

Datastores

Elasticsearch

<dependency>

    <groupId>org.elasticsearch</groupId>

    <artifactId>elasticsearch</artifactId>

    <version>0.90.3</version>

</dependency>

Jest

<dependency>

    <groupId>io.searchbox</groupId>

    <artifactId>jest</artifactId>

    <version>0.0.4</version>

</dependency>

MongoDB

<dependency>
<groupId>org.mongodb</groupId>
<artifactId>mongo-java-driver</artifactId>
<version>2.10.1</version>
</dependency>

---

Parsing

Jackson

<dependency>

     <groupId>com.fasterxml.jackson.core</groupId>

    <artifactId>jackson-databind</artifactId>

    <version>2.2.3</version>

</dependency>

StarQL

<dependency>

    <groupId>com.lithium</groupId>

    <artifactId>dev-ldn-starql</artifactId>

    <version>1.6</version>

</dependency>

org.json

<dependency>

    <groupId>org.json</groupId>

    <artifactId>json</artifactId>

    <version>20090211</version>

</dependency>

Jsoup

<dependency>

    <groupId>org.jsoup</groupId>

    <artifactId>jsoup</artifactId>

    <version>1.7.3</version>

</dependency>

Jparsec

<dependency>

    <groupId>jparsec</groupId>

    <artifactId>jparsec</artifactId>

    <version>2.0.1</version>

</dependency>