Storage Interface

The requirements for this project call for some type of interface which will allow clients to access storage services in a uniform way. On some occasions the client (e.g., on a smart phone, tablet or desktop) might want to store an object locally while on other occasions the same client might want to store or retrieve the same or other objects using a remote storage system.

Depending on the location of the client and the storage server and what connects them, different protocols may or may not be available. For example, one of the simplest and most robust methods to connect them is to use TCP / IP sockets. They are fast (the HTTP / HTTPS protocols used in the internet are the building blocks of TCP / IP). The issue with such approach is that access outside a facility requiring opening external IPs and ports is not desirable due to malware and hackers. Another simple approach might be to just file descriptors / file pointers (i.e., POSIX) but that would entail having file systems exposed externally. In additions different clients will be deployed on different software platforms (i.e., Android, iOS) so additional software would be required to interface them.

The idea would be to specify some type of connection type based on the decision made by the client to store an object on local or remote storage. Once that is done, the calls to store, query and retrieve would operate the same. This way the client could use the local file system or a RESTful API to store, copy, query or retrieve objects.

In addition, some type of access control to the storage system will be required. Given availability and popularity we will consider for this project OAuth 2.0 and OpenID. Both are open source so we will be able to address issues with them if needed.

All data should be encrypted while it is being transferred to storage and decrypted while it is being retrieved. Encryption at rest seems to be supported by most (never say all) file systems in the associated software platforms. The same holds true for network transfer protocols.  Let’s not reinvent the wheel if possible.

I always like to start with an initial architecture diagram. After that I like to follow with design (simple diagrams with very little wording), testing and implementation (the order is not a typo, always start with tests, which will initially fail and proceed until all is working). Keep in mind that the actual source code is always the final source of documentation for any type software. Always aim to develop clean, robust and well documented source code.  Such approach will help you and others validate the architecture and design allowing for changes before thousands or in some cases millions of LOC have been written :o(

Let’s start with the following architectural diagram:

Please note that this post should be referred to as a white paper. The final approach and implementation of the different classes will not be covered in this or any subsequent post related to the storage server.

Also note that in any system a mechanism to log activity like traces, warnings and errors is required in order to help maintain the software system. I will not cover such important aspect in this set of posts.

If you have comments or questions regarding this or any other post in this blog please leave a message in the area following the text for the post. I will reply as soon as possible.

Enjoy;

John

Follow me on Twitter:  @john_canessa.com

Leave a Reply

Your email address will not be published. Required fields are marked *