The Company Blog

Eric the Amazonian

January 22, 2010 · Filed under Careers, Uncategorized

Recently we have been doing work with Amazon S3 for scalable storage of assets we use in some of our applications. These files have been sitting on SAN disks and served out via web servers. It’s high performance storage dedicated to mass file storage. In all there is around 18TB of data and billions of files.

The inertia of moving these files to he cloud has been overcome by Amazons AWSImport. It’s a pretty simple concept, ship them a drive with your files and then have them import these files into Amazon S3 Buckets. It’s snail mail – at terabyte scale.

Firstly to cover the service, it’s simple and it works. If you grab S3Fox and install it with Firefox the whole process can be done in the browser. Step 1 is to create a manifest.txt which is basically a set of attributes for the ‘job’. The manifest defines the destination bucket, device ID (we used the device serial number), access control and also where to ship the drive back to.

With the manifest file you email AWS Import with a subject line of ‘CREATE JOB’ and the manifest.txt attached to the email. In generally less than a days time AWSImport comes back with a Job Number. You load this job number into S3FOX and you can create a SIGNATURE. Place the SIGNATURE on the hard drive you are sending to AWS and then ship it to the address. Very simple stuff.

Simple things still fail As we were sending alot of drives to AWSImport there was one we (read I) mixed up the manifest.txt and SIGNATURE. Luckly AWSImport has a CHANGE JOB functionality. You can email AWSImport with the job id and subject line CHANGE JOB and you can apply new manifests and signatures to the job. This saves you having one shot at goal.

We also found the AWS team were flexible on modifications to manifests and import processing. A good example is we needed Content-Encoding set to gzip for all imported files (as the majority were gzipped) and then we wanted to use the API to change it for a subset of files later. Within about a couple of weeks the AWS team added setContentEncodingForGzFiles:Yes attribute for manifest files (Thanks Eric@AWS).

We are based in Australia so when were last in the US some of the team went to Best Buy and bought a number of 1.5TB esata drives. This gave us the US power plugs that Amazon AWS required. The second mistake we made was forgetting to pack the power cord (duh!). Now this is where Amazon customer support starts to scare me with regards to how good it is. Put yourself in this situation with any online/offline company you deal with. You courier a drive to them for processing and you forgot to pack the power cable and they don’t have a converter. You then then get an email that looks like this.

“DELAYED JOB” – That’s not good. I was getting visions that the drive is coming back to me, I have lost cash on the courier and then lost time on the turn-around for getting data ready for production (worse than the cash). But then check out how Eric handles the situation.

Very Classy. It’s like I am dealing with a supplier down the road and when we (not I ) messed up instead of shooting me he saved me. I especially like “please let me know if this is unacceptable”. Eric even gives me the option if I don’t want to be saved!!

I wonder how any of the other cloud providers would have responded (even if they had a service similar to AWSImport)? I wonder how many of them have found employees like Eric.

These guys (through Eric) are killing it.

Leave a comment »

It’s not you it’s me (Internet Explorer 6)

November 5, 2009 · Filed under General, Incite Keystone, Updates

When we designed and built Incite Keystone we were aware that Internet Explorer 6 accounted for around 30% of our users. Due to this we have spent a lot of time customising the user interface to make it IE6 compliant – a lot of time. It occupies our designers, our front-end development teams and also our testers. This week we made a decision to not actively support IE6 anymore and talk to our customers about reasons to upgrade to IE7, IE8 or Firefox.

I could discuss how old IE6 is (8 years!!) and how you don’t have any other software on your computer that is 8 years old. I could also discuss the numerous security/vulnerability issues in IE6 however this would probably be construed as spreading FUD (Fear, Uncertainty and Doubt). So instead of that I decided to try and present an argument based on straight speed. After all you’re the person using Incite Keystone all day. You more than anyone would understand that if you could reclaim some of your valuable time, that would be awesome.

So I looked back across our logs and decided to build a virtual machine that matched the majority of our users. Windows XP accounts for 76% of our users followed by Windows 7 (9%) and Windows Vista (8%). Yes the Windows 7 result is interesting. As for browsers IE7 makes up 46% of all browsers and IE6 accounts for 26% and IE8 18%. It is this 26% of our IE6 users that need the speed! We use javascript extensively as do all other Web 2.0 applications. So javascript execution speed is important for the overall speed of the Keystone application.

I built a Windows XP virtual machine with 512MB of RAM, 1 CPU, IE6 and Firefox 3.5.4 and tested the browsers against the Webkit Javascript Benchmark. Follow the link to do some testing yourself and compare.

IE6/512MB RAM #1 - 72921.4ms

IE6/512MB RAM #2 – 65049.4ms

IE6/512MB RAM #3 – 77139.8ms

FF/512MB RAM #1 – 1167.8ms

FF/512MB RAM #2 – 1297.4ms

FF/512MB RAM #3 – 1177.4ms

The results look unbelievable but that is the difference 8 years makes! For the sake of argument let’s take the best result for IE6 (65,049.4ms) and compare it to the worse result of Firefox (1297.4ms). The result is IE6 is 50.1 times slower than Firefox on the same hardware.

Bank some daily time savings and move away from IE6, but should you move to IE7 or IE8?

When I tested IE7 and IE8 they were 1.39 times faster (46902.0ms) and 12.1 times (5383.6ms) faster than IE6 (respectively). But note still not even close to Firefox. When you compare FF to IE8 it is 4.15 times faster.

When I say we won’t be actively supporting IE6 it doesn’t mean we will try and break things, just that we won’t be testing IE6 against Keystone release versions from now on – so mileage may vary. However despite this I think this blog post alone and your own testing will prove to you that it is time to break up with IE6 and move on.

Comments (1) »

Toolbox Drag and Drop Upload

September 24, 2009 · Filed under Incite Toolbox, News, Updates

We were going to wait a couple of weeks but drag and drop upload is so cool it deserved a deploy and a blog entry. If you are happy to install the Yahoo Browser Plus plugin, when you login to Toolbox the page will be enabled with a drag and drop zone.

Drag and drop some files into the area and they will automatically start uploading to Toolbox. It’s a subtle touch allowing Toolbox to operate more like a desktop application.

Once again if you like it let us know on Twitter @incite_toolbox and if you want to see it in action check out the video.

Leave a comment »

Incite Toolbox Beta

September 22, 2009 · Filed under General, Incite, Uncategorized, Updates

It has been awhile since I blogged and one of the big reasons is we have been busy building a new project called Incite Toolbox. As a company we build project collaboration applications; the most well known is Incite Keystone.

I was intrigued by the idea of a fast, real-time, collaboration product that small teams could start collaborating with in minutes – no setup, no training. Incite Toolbox embodies the basics of our initial idea. Creating a project takes seconds and you are collaborating with your team instantly.

For messaging we figured you would want to talk or post to the Project and also talk privately with individuals. It has a feel not dissimilar to Twitter with the additional capability to send files as well.

A multiple upload functionality allows for files to be easily added to the Project. Add an image and it is thumbnailed and all files are scanned for viruses. If you don’t want to log into the web interface then just send your files via email by placing the project email address in the TO: or CC: field.

The wisdom of crowds can be harnessed through the use of tags for messages and files. We will extend this further in the future but for now tags give you the ability to rapidly sort information. You can tag, your team members can tag and everyone enjoys from the group classification of data.

One of the features I enjoy using is the instant search. It is extremely fast allowing you to sort 100′s of messages instantly. It uses a scoring system so that the more accurate your term the more accurate the results. It’s something you need to try so you can see how cool it is. Also because of the technology we use there is no round-trip to and from the server so it fast – very fast.

Incite Toolbox supports multiple languages out of the box. This is a little sketchy at the moment but worthwhile to play around with. Change your language from English to Chinese and you will see some (not all) of the menus change. Then change to French and then to German. So even in this very early preview you can see the dedication to internationalization.

Lastly you can subscribe to SMS or Email updates. Be careful with this it is very *spammy* at the moment. However in the future you can watch tags or project members and receive updates when they contribute to the project.

I would also like to see what you guys think of Toolbox – so feel free to follow/tweet you’re experience with it. Just be kind, remember it is very early days.

Follow or Tweet – @incite_toolbox and also a Video! and don’t forget to try it out and let is now what you think!

Leave a comment »

Incite S+S Architecture @ Remix

June 18, 2009 · Filed under General, Incite

The video of our session at Remix is available on the web. You can find it here.

Leave a comment »

Listface available on Github

June 16, 2009 · Filed under General, Incite Keystone

Users love Keystone’s auto-complete when addressing messages to users on the system. Not that we can take credit for the concept but what we can take credit for is a cool jQuery plugin that our Rails team wrote and have made available on GitHub.

Now there are a couple of similar projects kicking around (Facelist for example) but the license didn’t suit our product (commercial) so we wrote Listface.

Check out Listface here.

Leave a comment »

Keystone – Outside the Browser

June 16, 2009 · Filed under Uncategorized

At Remix we demonstrated a typical construction process that normally would take 2 or more hours to complete - we demonstrated how it could be done in 2 minutes.

A picture of a road defect was taken using a HTC Windows Mobile 6.5 phone. Embedded within the image was EXIF data that contained latitude and longitude information.

The defect picture is selected and attached to a form using the gesture-enabled WinMo 6.5 device. A defect form was downloaded from Keystone with all properties and attributes including what properties were mandatory (shown in red). Tsarina filled out the details and submitted the defect form making it immediately accessible within Incite Keystone.

Incite Keystone Mobile @ Remix

Tsarina then assumed the role of another Incite user back at the office that was acting on the defect. As can happen in any complex project the defect may or may not be an actual defect. Within Incite Keystone’s full jQuery/AJAX interface the defect form was accessed and then forwarded to a particular sub-contractor (played by me) as a Request for Information (RFI). Tsarina then attached a design specification to the RFI for the defect to be checked against.

The goal of the demo for us was to demonstrate how Incite Keystone can be used outside of the browser – giving the customer a series of tools and allowing them to find their own way of doing things. Instead of checking the RFI in a browser all files linked to the RFI were downloaded to my notebook using Microsoft Sync Framework. An icon in the Windows 7 taskbar displayed the progress of this download then pulsed.

One of the features we liked about Windows 7 was that you could right click on icons in the taskbar and get a Jump List. This allowed us to add multiple actions to the icon – you can open the original RFI, the defect image or the design specification that was added to the RFI.

Clicking on the mail icon opens the RFI message in Keystone. The defect image and the document were synchronised to my laptop so I can open them up from my local hard drive using the Jump List.

In the demo I checked the defect image, then opened up the design specification via the Jump List. This allowed me to show the work we had done with Microsoft Office and how once again a user could work outside the browser.

Incite Keystone’s Office Ribbon allows users to Search, Edit/Update documents and also form related information (e.g. Defect, RFI etc) from within Microsoft Office. The Incite Ribbon header shows all the options available to the user.

By clicking ‘Search’ a panel is rendered on the right of the document being reviewed and you can search across all text within documents in Keystone.

Double clicking on any document returned by the search downloads the document and opens it within Microsoft Word. This can easily be uploaded back to Keystone as a new document revision by selecting Update Form.

Updating the form allows an Incite user to modify important meta-data prior to saving the document within Incite Keystone.

Lastly the demo was concluded by displaying the original Defect on a high-resolution aerial photograph of a roadway which had been stitched into Bing Maps (formerly Virtual Earth). This allowed me to zoom into 10M above the roadway as opposed to Bing’s 200M zoom level. Zooming in and out was awesome due to the Silverlight Map Control for Virtual Earth. Within the Keystone Office Ribbon I selected ‘View on Map’ and was able to explore Incite Keystone form data spatially.

Leave a comment »

Code from Remix

June 11, 2009 · Filed under Incite Keystone

Today at Remix 2009 we demo’d the Silverlight Virtual Earth Mapping Control and also our application Incite Keystone Mobile running on an Windows Mobile 6.5 device. The code from the Silverlight/VE demo has been contributed to the Deep Earth project. The demo itself can be viewed here. I must also thank John O’Brien for his help with the mapping demo! Check out Deep Earth as well, I love that project.

The Windows 6.5 application we showed had full gesture support. It was developed before the WM6.5 Toolkit was released and required us to wrap the API’s so that we could use them in managed code. This work we contributed to the Mobile Contribute on Codeplex. Thanks to James McCutcheon for helping us with this code alongside our resident mobile guru at Incite.

Incite Keystone Mobile @ Remix

Leave a comment »

Remix 2009

May 14, 2009 · Filed under Incite, Incite Keystone, News

Incite Keystone sits on a Microsoft stack and so over the last year we have worked closely with Microsoft on particular areas. This gave us a chance to keep them updated on the cool/advanced things we were doing with the .NET 3.5 Framework, Silverlight, SQL 2008 etc.

As I have mentioned before most of us at Incite are a little self-effacing when it comes to talking about how good we are. The Product Development team are awesome designers and coders, but we aren’t trying to tell the world – well until now.

Last week we accepted an offer from Microsoft to participate in the Remix 2009 keynote and also take a session. I like the idea of Remix, laid back, geeky and the potential to wear jeans and a t-shirt whilst presenting On top of that we get to demo some of the cool things we have built and have a look at some future ideas we are working on.

Yes I am already praying to the demo gods (fingers crossed).

You can check out the details of the Keynote and Sessions for Remix here.

Leave a comment »

New archiving options

April 30, 2009 · Filed under Archive, News, Support

Incite provides ‘online’ and ‘offline’ archive options for customers when their project has been completed or when their role on a project has finished.

Online Archive
An online archive involves changing the project to read-only mode. It is still hosted by Incite in the cloud however it sits in stasis. Users can log in but they can’t modify any data.

Offline Archive
An offline archive involves extracting a customers information from Incite and making it available as a copy via a direct download or USB Drive. In the past offline archives were produced via a manual process – creating filters and saving the messages, documents and files associated with these filters. This led to lengthy processing delays when a project requested an offline archive. The other issue related to the old archive method related to the structure of the data returned. All messages and documents were returned (in PDF) and attachments were returned as the original files. However no contextual information was retained. For example there was no way to navigate or find information relating messages to documents or files.

We decided to automate the process and develop code to allow us to automatically extract all data from a project and not have to rely on the old manual process. In doing so we created two forms of offline archive – Standard and Advanced (not very imaginative I know).

Both the Standard and Advanced archive process extracts all data based on Incite’s role-based security. This ensures only data that your company’s users have sent and received is archived.

Standard Archive

The Standard Archive provides a directory structure organised by Document Type/Year/Month/Day/ then the Time and Name of the sender. Within each directory is an XML file (message.xml) which is the precise data from the message or document, a message copy if your project is configured with them and any attachments contained in the message or document.

Although the message.xml file provides relational information between a message/document/file a customer would need to parse the XML to rebuild this contextual information.

In addition although Google Desktop or some other search software could be used to full-text-index the archive it is something performed by the customer once the archive is received.

The standard archive can be processed quickly and made available as a download over the internet in what we refer to as an Express download option. Express download involves the archive being directly downloaded via the internet rather than being sent to the customer on a USB disk.

Advanced Archive
The Advanced Archive uses Microsoft Sharepoint Services to provide full-text search capabilities and maintains all contextual information between messages, documents and files. Project data is extracted from Incite and injected into a Microsoft Windows Sharepoint website (Windows Sharepoint Services 3.0).

We selected Windows Sharepoint Services (as distinct from Microsoft Office Sharepoint Server) as it is freely available to customers who have a license for Microsoft Windows 2003 Server. It does not require you to purchase Microsoft Office Sharepoint Server but ofcourse if you already have a Sharepoint server you can import the archive into your existing site.

The Advance Archive is delivered as a database file on a USB hard drive. Accompanying it is a PDF document detailing how to setup the Windows 2003 server to import the archive.

The Advanced Archive has a number of advantages. Relationships between messages, documents and files are maintained and you are able to navigate through the system using these relationships. In addition you can create views within Windows Sharepoint Services in a similar way to filters. Lastly the entire archive is full text indexed so you can search across messages, documents and text within files such as Microsoft Office documents and PDF’s.

Customers who are interested in an archive should contact support or their Incite Account Manager.

Leave a comment »

« Previous entries

The Company Blog

Eric the Amazonian

It’s not you it’s me (Internet Explorer 6)

Toolbox Drag and Drop Upload

Incite Toolbox Beta

Incite S+S Architecture @ Remix

Listface available on Github

Keystone – Outside the Browser

Code from Remix

Remix 2009

New archiving options

Search Blogs

Post Archive

Post Categories

Tag Cloud

Recent Tweets

Follow “The Company Blog”