See the following screenshot for a detailed folder views.

I know this is all kind of dirty, but it does its job. As improvement I am waiting for an OS X version of rsyncrypto which should be more efficient in encryption than murk. I’ll keep you posted.

As a followup for my local rsync script compilation I now present a small python script which takes care of rsyncing your folders to your S3 buckets and taking care of local encryption. I am really fond of the idea of “storing your data in the cloud”.

You will need:

1) An Amazon AWS Account with S3 enabled, sign up here. Amazon will charge you 0.15US$ per GB stored per month.

2) An Rsync Account from s3rsync.com. They built a special rsync setup in an Amazon EC2 cloud in order to allow real partial file syncing. The problem with direct rsync-S3 connections is that S3 only allows you to PUT entire objects (files). They charge you 19US$ per 380 rsync-hours which is 0.05US$ per hour. Remember, only transfers after huge changes in your data structure are time intensive (and of course slow connections). See their FAQ.

You will also need to install some tools on your system in order for the encryption to work properly. The special issue with rsync encryption is, that normal encryption usually messes up your entire file structure and therefore the entire file needs to be transfered at all times. Rsync would not be of much use in such a case. Although Murk weakens encryption by dividing a file in small blocks and encrypting each one of them, it seems to be the best transfer-efficient solution.

Be sure to have your developer tools installed and first get the following two source packages:

- BZip2, compile with ./configure && make && make install

- OpenSSL: ./configure darwin-i386-cc && make

Then get the Murk sources and compile them since you now got the relevant crypto and compression libraries.

- Murk: ./configure -L ../openssl-0.9.8h && make && sudo cp ./murk /usr/bin/

Next create a new cipher key so Murk can de- and encrypt in standalone mode. I suggest using AES-256: murk -a <your_key> -c aes-256

You would now be able to compress a folder, e.g. test and encrypt it in an rsync friendly manner by doing: tar -cf - test | murk -n -c aes-256 -k /Users/fluppel/.murk -v -o rsync.tar.zm <your_key> However I learned that piping the tar output to murk produces corrupt files, at least with my setup. Try using a tmp file by issuing the following commands: tar -cf no_cipher.tar ../test && murk -n -c aes-256 -k /Users/fluppel/.murk -o rsync.tar.zm <your_key> no_cipher.tar && rm no_cipher.tar

The *.tar.zm file can be decrypted by using: murk -n -k /Users/fluppel/.murk -d niquille rsync.tar.zm

This technique now allows you to encrypt large directories and rsync them to an S3 bucket by using the s3rsync.com service. But before issuing the command, be sure to save the supplied SSH private key to a local path of yours. Then issue: rsync -v –stats -e “ssh -i /Users/fluppel/.ssh/s3rsync_dsa.priv” -az /Users/fluppel/Desktop/Inbox/test_due/ <your_user>@farm.s3rsync.com:%%<your_bucket>%%<your_AWS_id>%%<your_AWS_secret_key>

Now the nice thing about this solution is, that your backuped directories reside in a bucket somewhere in an Amazon datacenter in the US. You can access the *.tar file anytime by using Cyberduck, JungleDisk or s3fs. You can then just download the file, uncompress it and decrypt its contents by using the murk command as described above. Remember, now you will pay transfer fees to Amazon (whereas by using the rsync service you do not pay traffic but the hours used). I get around 250 - 350 kbits/s downloading from a US S3 bucket to a european location which is not so bad.

Now this all seems pretty complicated. I therefore compiled a small python script which actually takes care of encrypting certain parts of your data tree and rsyncs it to your buckets. I personally do not encrypt all my data for two reasons:

1) Your data is being transfered over an SSH connection and is already being stored in an encrypted bucket by Amazon.

2) Encrypting my whole documents folder and so forth always uses a great amount of CPU and time. I therefore limit encryption to more or less sensitive data. In the end it really all comes down to how much you trust your gateways and hosters.

Feel free to use my python hacking. I know its not perfect but I think it will do its job. Try calling it from crontab or whatever service you like.

There is one issue with the script though: The rsync exclude pattern does not seem to work properly with folder names containing whitespaces.

Pev’s tip though finally appears to work on 10.5.3 (I remember trying on 10.5.2 without any success). Enter sudo ifconfig en1 mtu 1492 in your Terminal and try resetting the pram.

I think doing the following in your System Preferences -> Network -> Airport permanently sets your MTU (surviving restarts).

Enjoy (and let me know if it works).

update: The Django crew now finally announced a version 1.0 coming in September this year.. Some people have problems to accept the slow (!) Django release process.

This slide shows a simplified illustration of a cloud platform setup. As you can see virtualization software actually allows a certain service to run on multiple physical entities, regardless of their geographical location or quantity.

I later-on introduce my hypothesis, that the industry is now reaching a third level after in-house IT solutions, later outsourcing and now the cloud (see slide 6).

ZDnet also has a nice comparison of Google App Engine and Amazon Web Services.

John Willis wrote a good post about the demystification of the cloud and introduces an interesting “layered” classification of cloud services. Also see his A-Z list of current cloud-providers.

some excerpts ..

• Urs: 
  • “The internet is like a cake. You have the biscuit (physical layer), chocolate (logical) and cream (content)”
  • Four different modalities of regulation exist:
    • self-regulation of markets
    • law (regulation e.g. IPR)
    • social norms
    • code (such as country specific google search results)
  • I get hunders of messages a day, spam doesn’t annoy me that much, although there is student spam..
  • interoperability results in more competition and therefore more innovation; interoperability becomes key when talking about national innovation policy; however disruptive innovations theoretically have greater impact without regard to interoperability
  • cyberspace solved classic market failure of information asymmetry, but how should limited processing capacity cope with large amounts of information and how can information quality be assured (also from a regulatory perspective)
• John:
  • “ISP’s are a wonderfull place for regulation”, meaning it is the best spot for law-enforcement from an architecture point of view.
  • Talking about Spam: Why not change the paradigm of the web (the accountable web). Instead of accepting everything by design one could actually reject everything by default and rely on what trusted “friends” accept.
  • Bill Gates wasn’t able to solve the spam problem (although he was confident to have it solved by 2006) -> Gasser: he should have said one wouldn’t care so much anymore.
  • “Spam is not just about putting the Spam Kingpin to Jail”
• Eric (Student):
  • We get Robocalls (spam) in India on our cells

„Computing is turning into a utility, and once again the economic equations that determine the way we work and live are being rewritten.“

Today’s internet age is both vibrant and old-fashioned in economical terms. Every day new internet start ups sky rocket. Others disappear as fast as they built upon the vast potentials. But despite new technologies being the enablers of such potentials, success is by no means guaranteed. That’s where old-fashioned economical and managerial principles come into play. One such example is the TV broadcaster Joost, which had been founded back in 2006 by the founders of Skype. The company developed a set of video compression algorithms and network protocols in order to supply viewers with free peer-to-peer high quality video. Despite the working system, Joost failed to integrate up to date content, especially live tv feeds into their broadcasting network. Recent rumor has it, that the company is undergoing a major restructuring and fallback to US markets (instead of worldwide). This is where the competitor Zattoo shows how to be successful. Although Zattoo relies on a different video codec with weaker video quality, the company managed to attract a broad audience and place targeted advertisement. Interestingly Zattoo uses a similar peer-to-peer distribution technology. One would be fast in concluding, that technology itself doesn’t define a company’s success. As Manyika states, “technology alone is rarely the key to unlocking economic value: companies create real wealth when they combine technology with new ways of doing business”.

This paper highlights technologies underway on the internet that could have high economic impact on certain areas of business (on the micro level). In search for the next determining building blocks for the next generation of digital products and services, several aspects of economic impacts of these technologies will be discussed and analyzed. However, this paper will not cover every technological detail and economical implication in-depth; it is meant to give a brief overview and create a basis for discussion.

The report is divided into four major business concerns, each covering the most related technologies and their anticipated economic impact. This reverse approach was chosen due to the focus on business relevance.

Scalability as key growth factor

One of the year-to-year and long-term strategic challenges of a company is growth. If one talks about an Internet start up such as YouTube (now part of Google) back in 2005 and considers its success, company growth meant increasing market shares in the market for online videos. This specific growth in the user base meant an ever-increasing need for storage capacity, processing power, network bandwidth and higher overall loads. A couple of recent technologies address issues such as scalability, which is one of the key drivers of success when talking about growth. This section will first briefly cover these technologies and then assess their specific impact.

Data 2.0 follows Web 2.0

Although the term Web 2.0 stands for a huge variety of recent phenomena, not only YouTube, one of the post-bubble winners, has shown us how drastically data amounts can increase. When talking about data from social networks such as Facebook, search indices at Google or Yahoo, customer records at salesforce.com or link records at digg.com, data complexity also increased. This fact will have a huge influence on database management. Nevertheless, companies such as Amazon or Google have expanded their business models and enable third parties to use their infrastructure in return for a usage fee. Amazon discovered that they had such high capacity margins at their data centers, that around 90% would normally reside unused. Due to their long history of providing reliable in-house web services, Amazon introduced SimpleDB, EC2 and S3 services. Google just recently introduced Google App Engine as public beta, which is a combination of storage solution, BigTable Database and Webservice. Companies such as YouTube would now be able to use these data centers as ultilities, smiliar to plugging your fridge into the wall and consuming electricity on a pay-as-you-use-it basis (and therefore grow at a pace previously not possible). Although these concepts of grid-computing are not all new, the commercial usage and feasibility has only just recently become possible. This is due to large advancements in hardware performance (also see virtualization) and the usage of standard protocols. Services such as those from Amazon or Google are considered disruptive and initiate a change in the world of relational databases.

Virtualization

Similar to Google, Amazon uses virtualization technology to serve different customers with their unique set of applications and data (EC2). The technology allows multiple systems on the same hardware due to an inserted layer of software in-between. The possibilities are enormous, since the operation system level becomes hardware independent and therefore scalable across multiple physical systems. This is the foundation of utility computing.

From an economical point of view these advancements in database systems and hardware usage allow a more efficient use of fixed assets. Providers of data centers or experts such as Amazon (supply side) are able to reduce total costs of operation and therefore leverage their hardware. The shift towards utility computing in outsourcing eliminates the traditional client/server model and actually allows greater economies of scale.

Outsourcing not only becomes a bargain for suppliers, it also gives the customer (demand side) full flexibility (especially in terms of speed) and, most importantly of all, scalability. The unbundling allows a company to free itself from large fixed investment in hardware (and the overhead) and minimate risk by achieving a competitive marginal cost (only computing, storage and bandwidth usage is being charged). But great opportunity also bears risks: What if utility suppliers do meet physical capacity restraints (which do exist) and what if services have major downtakes? Despite encryption, businesses with privacy-sensitive data will have further concerns regarding utility computing.

An Attempt at coping with data flooding

Since the early dawn of the internet the amount of content has been growing at a fast rate. For more than a decade search engines have been aiding us with simple lookup tasks and pretending to structure the web, or better, it’s content. One must agree that Google and its competitors are quite good at searching written content, such as texts, URLs and Metadata. But this search for the needle in the haystack becomes extremely tedious if one includes additional data such as images, music files or even video. Following the idea of a semantic web, standardized metadata needs to be created in order for greater automation to succeed. But there are still the problems of how metadata gets created in the first place and how certain quality requirements can be met.

How to extract Metadata

Current image search technologies are able to identify objects within a picture and convert them into metadata tags. Examples are a Coke bottle appearing in a picture being labeled as coca cola or a tiger lying in the bushes labeled with tiger and gras. But problems persist: By including more complex lookups such as a search for “happy faces” or facial recognition with and without beard current algorithms come to a limit. Furthermore the spread of GPS devices has introduced geotagged pictures that include additional location specific information. Such metadata enables entirely new applications such as integrated maps and timeline services and location based clustering of images.

Syndication: gather information

As solid part of Web 2.0, RSS/Atom feeds allow the interchange of information across systems. Aggregation and filtering of multiple feeds enables advanced usage such as complex lookups which span across multiple sources. Yahoo introduced a service called Yahoo Pipes, which gives users the possibility to drag&combine different elements to a new syndicated feed (Forrester, 2007, p. 22). Think of a lookup, for example, for an apartment with a gym near a park in New York City, within a specific price range.

These current technologies and metadata standards allow decent structuring and filtering of text data and, up to some extent, images and sounds. Since employees spend hundreds of hours searching information and pursuing all sorts of research, it is clear that abundant metadata can improve this process. Taking a job recruitment process as an example, one is able to cut costs on one hand due to less service charges and transaction costs. Social Network sites such as Facebook or meta people searches already allow fast and more or less segmented search for potential targets. This step towards coping with information asymmetry does not only increase efficiency, it also widens the usable information base for a certain entity and increases transparency, for instance, for the job applicant. New possibilities such as geotagging and enhanced image search could allow companies involved in travel activities to offer even more integrated and localized services as added-value. Customers could for example compile auto generated galleries for their individual trips (location based services).

On second view, coping with information also offers greater exploitation possibilities in terms of running artificial intelligence algorithms against (meta)data. One of the leaders at early stages in this trend is Amazon. Analysis of purchase histories provides the recommendation engine with proposals for future customers. The information derived from these algorithms increases conversion rates and therefore has a direct effect on sales. Such information not only helps managers to make smarter decisions. It also lowers costs for example for customer acquisition due to deeper insights and enables targeted attention. Digging further into the use of (multiple linked) metadata reveals astonishing possibilites not yet present. - “Since we are essentially meta-tagging ourselves through our social networking memberships, shopping habits and surfing addictions, it’s conceivable that the information could attempt to find us” . - Such a drastic technological advancement in search could change the way companies promote products through digital distribution channels and again, improve customer satisfaction and at the same time reduce acquisition costs.

Another attempt at breaking down information complexity has been undertaken by NASDAQ Stock Market Inc. They use a newly introduced product bridging on- (web) and offline (desktop) usage called Adobe Air. The runtime allows analysts to run Nasdaq Market Replay, which computes several intensive statistical processes on a huge amount of data on their local desktop and get instant results. It is modern syndication (webservices) and a lightweight Flash/Ajax environment which relieves servers of high loads and merges different pieces into one application. Although this concept is at first sight contradictory to the concept of utility computing, there is space for a dual existence. Security concerns (as stated above) could force specialists to use local or limited resources, especially considering the financial industry. Furthermore desktop applications allow offline usage, which gives further flexibility to employees working en route.

Enriching digital human interactions

Since its development the Internet has survived without a profound security layer built into the core. Despite various existing “overlay” technologies such as encrypted data transfers via SSL or VPN, 802.x authentication, Public/Private Key encryption algorithms and Smart Card Authentications basted on TPM chips, identification of individuals (transparency), integrity and usability remains an alarming issue.

Digital Identity: Who am I trusting?

Current user-centric approaches such as OpenID focus on the bundling of the countless and tedious authentication and authorization routines needed in order for users to interact with various Internet services. It offers a unified login protocol (single sign on) and has positive network effects due to partnerships with industry leaders like Yahoo and Google.

Voice over IP: convergence?

Besides professional solutions such as those provided by hardware manufacturers like Cisco, eBay’s Skype has been a disruptive factor in online voice communication. Since the foundation in 2004 Skype has provided voice and video streaming over the Internet based on a closed protocol that incorporates peer-to-peer technology. Similar to SIP-Applications, Skype enables point-to-point streaming of high quality live voice and video feeds and most important, a gateway to worldwide telephone networks. Despite the enormous potentials, threats such as monitoring, eavesdropping or spamming exist due to the nature of building upon an IP-network.

Technologies that accelerate convergence between telephone networks and the Internet, such as the Skype protocol, dramatically reduce operating costs of modern communication channels. Thinking of the vast increase of negotiations and ad hoc collaboration across long distances in business, such modern technologies step in and allow greater productivity (due to lower transaction times) at lower costs (Skype allows global calls at local rates whereas Skype-to-Skype is free). Manyika takes it one step further and proposes the possibility of value creation through smarter and faster interactions among individuals, by also using other modern tools such as Wikis (p. 4). She furthermore outlines the potentials of “integrat[ing ]and manag[ing] the work of expanding number[s] of outsiders”, namely shifting the work towards scalable freelancers.

But despite the high potential productivity improvements, fraud and malware such as Spam show how important valid identities are in daily digital communication. Fishing attacks steal user credentials and abuse user data, credit cards in e-commerce transactions are subject to theft. Users on the other hand can not verify whom to trust in e-commerce transactions. E-Mail traffic without digital signatures does not allow a true verification of sender and recipient due to spoofing possibilites. As crucial as this issue is to business and interaction in business, fundamental design failures of the internet currently do not allow a unified solution. Still, technologies such as modern signing algorithms are able to dramatically increase identity awareness between point-to-point E-Mail, voice or document verification in daily business communications. One important aspect in business, besides security, could be the advancement in a trust-building process when considering customer-company interactions.

A promising maturing technology, especially from the standpoint of utility computing and abundant bandwidth due to fiber optics, is mesh networking. It describes ad hoc wireless coupling of multiple devices which instantly build up their own network at their current location. This technology unfolds its true potential for applications in remote areas due to its flexibility and robust self-organizing structure. It not only drastically lowers investments (e.g. by building a temporary network); it also allows new applications, such as ad-hoc sensor networks, to be implemented.

The rise of new business models

Modern technologies such as those mentioned above do not only have implications for existing (internet) businesses. As already stated, they also unleash a tremendous potential regarding new ways of doing business – it is modern business the cyber way.

Virtual Reality: Where The Matrix starts?

The most successful example of a functioning sort of duplicate of our world is Second Life, which has been in development since 2003 by Linden Labs. Utility computing allows real time computing of a virtual world with thousands of users actually interacting within that space. On average around 30’000 to 40’000 unique individuals talk, sell properties, build houses and enjoy virtual life. Second Life has its own currency called Lindon Dollars, which can be bought in exchange for real US dollars on Lindon FX. Total supply of Lindon dollars as of April 2008 was about 4.8 billion, which equals around 18 million US dollars.

Second Life was built as a scalable system from the beginning on. Similar to computing clouds such as those offered by Google or Amazon, Lindon Labs runs Second Life as a real time streaming system on their cloud called Second Life Grid. End-Users access the system by downloading a tiny thin client, that allows the rendering of the individual view of the 3D world on the users access client.

Peer-to-peer data exchange: BitTorrent

Nowadays a significant portion of Internet traffic is accounted for not only by video streaming (such as by YouTube or Hulu) but mostly by peer-to-peer traffic using the BitTorrent protocol. It was developed in 2001 and is built as a pareto-efficient distribution system due to robust resource utilization and cooperative techniques. The protocol bypasses a typical HTTP limitation, which is a common upload-bandwidth limitation. By incorporating a centralized tracker and peer concept, peers download small chunks of data from other peers. The protocol builds upon network scale effects, meaning that more peer-participants actually increase, rather than decrease overall performance, therefore eliminating the mentioned upload-bottlenecks.

Similar peer-to-peer concepts apply for the already described voice over IP technology behind Skype. Users generally contribute without even knowing.

The transfer technology BitTorrent and other peer-to-peer concepts become even more disruptive, if one has a glance at what actually gets transferred by peers. More than others, the music and film industry see themselves confronted with drastic decreases in sales of CDs, DVDs or online products such as on-demand video. One hypothesis could be, that a shift towards illegal channels such as booming portals like thepiratebay.com, has driven away demand from their traditional business models. This implies, that technologies such as BitTorrent not only are a threat to traditional channels, but actually underline the urge for the creation of new business models. They should therefore incorporate these new ways of customer behavior (e.g. on-demand, cheap) together with technologies that enable modern ways of distribution and revenue generation. A successful example within the same industry is Apple’s iTunes Music Store.

What appears to be compelling about virtual realities such as Second Life, is that humans are actually willing to spend vast amounts of their time in an artificial world built out of 3D graphics, modern physics and weather algorithms as well as a virtual market place co-created by other real humans. As a PEW study suggests, these worlds are meant to become more and more important and should lead to higher productivity for “technologically-savvy” communities. Linden (among other competitors) demonstrates how to incorporate virtual layers in new business models. By creating a world with replicas and even new variations of markets (such as housing, shops or attractions) such a company creates new revenue possibilities (by offering a gateway to a virtual layer in exchange for fees) never seen before. Most interesting is the fact, that existing companies such as Gucci or Prada see themselves confronted with both new problems and challenges. Copy infringement in Second Life is becoming a huge issue because people are vastly creating digital counterfeits for Avatars. But thinking of a company such as Prada opening a real store within Second Life attracts a whole new opportunity of distribution channels for (new) products. Furthermore IBM just recently started an extensive partnership with Linden by having an eye on future business applications: “The 3-D Internet is a transformational opportunity that will change many industry processes, gather new revenue streams, and increase productivity and brand opportunity.”

Conclusion: Utility Computing and Business Model Adaption

This paper highlights several technological developments in brief and analyzed several economic implications regarding each of the four big business topics. First the concept of utility computing was introduced in order to show growth potentials for both supply and demand sides of computing clouds. The concept implies revolutionary data storage technologies optimized for speed and reliability. Second, the question was raised as to how current information flooding could be structured and reversed into useful data. The ideas of a semantic web building on top of metadata and data interchange based on syndication technologies were discussed. Modern applications such as Nasdaq Market Replay show how such information can be applied and build upon a desktop/server model (at first sight in contradiction to utility computing). Third, the paper briefly reviewed advancements in digital identity handling and modern peer-to-peer based communication technologies such as the one used by Skype. Possible improvements in productivity and the importance of trust in interactions are being discussed. But most importantly, these technologies show how far convergence has come. Finally, virtual worlds and efficient transmission technologies show the need for existing businesses, especially the music industry, to rethink current business models and create new, innovative ways of revenue generation.

This brief overview does not just emphasize how important information technology has become for a wide variety of businesses. What is most important is that it demonstrates how some disruptive technologies are real opportunities to cut down huge chunks of fixed costs and revolutionize the way of doing business as well as interacting with stakeholders. It is not far fetched that Carr suggests: “In the long run, the IT department is unlikely to survive, at least not in its familiar form. It will have little left to do once the bulk of business computing shifts out of private data centers and into ‘the cloud’.”

BTW: Wireless ist still heavily broken and Apple didn’t seed 10.5.3 final yet.. That’s why I started to lay out old fashioned cables again.

I saw my iPhone using up all its battery power in less than 12 hours. The solution: Navizon GPS 2.0.2 appears to run some daemon which eats up all the power.

By the way, this Zibri guy who builds ZiPhone really is some kick-ass dude. His ZiPhone 3.0 and so forth really takes away all the hassle of downgrading, upgrading, patching, although it was fun. (It’s so easy, a caveman can do it; even all my friends who don’t have a clue can use it..) He also appears to have found a way to activate the beta 2.0..