What is Twisted Storage?
The market speak is:
An open source "object storage" software platform to manage the placement, protection and processing of large quantities of fixed content in a massively scalable, extensible grid architecture based on commodity hardware and legacy storage.
In non-market speak Twisted Storage is an application that will take your data and manage it for you, just like a file system. But Twisted Storage goes way beyond what a normal file system does.
What types of "objects" are stored in Twisted Storage : just regular files. The data stored in the system is expected to be fixed content, meaning it doesn't change at all. If you read the data back, change it, and store it, it is considered a different file. There are plenty of examples of fixed content data : emails you received, movies or mp3 files you made or own, documents you wrote, etc. In fact it is estimated that 80% of the data you have on your computer is fixed, reference data.
Your Twisted Storage system can start out small - a single computer with a few disk drives. When you need more storage you can add more disk drives and more computers. But it has been designed to store petabytes worth of data without requiring any special hardware. For example storing a petabyte worth of data, using the biggest disk drives available today (500 GB), would require 2000 disk drives. To overcome the need of special hardware, Twisted Storage is a scalable, distributed system using ordinary commodity computers. Even so that petabyte of data would require 500 computers networked together and that requires special handling.
Twisted Storage has a few features you won't find in any file system. For example once a file is added to the Twisted Storage repository it can be written in multiple locations simultaneously. The system has been designed to be compliant with Sarbanes-Oxley Act: when an object is added to the repository you can no longer delete it and the integrity of the contents are guaranteed. In addition any access to the object is recorded.
Twisted Storage is designed to not need backup or recovery programs: the system is designed to automatically recover data that has been lost. The system is designed for ease of management: any change made to any system in the repository will be automatically rippled to all the other parts of the ssytem. If some system should lose a file, that system will automatically locate it on another system and download it.
A Twisted Storage repository is also capable of processing data on each of the systems that make it up. Rather than writing an application that runs on some server outside the Twisted Storage respository, reading data, modifying it and sending it back, you can easily run the application on each computer that holds the data in question. In other words the programs run on the repository servers.
Unlike the other fixed content systems (the rest of the world calls them CAS systems - content addressible storage) Twisted Storage can also support legacy storage devices. All the other systems out today require "green field" applications: you have to start from new, empty systems. Twisted Storage is different: it can use existing storage. There exists an interface to EMC's Centera product; as time goes on new interfaces will be added.