I do not like to live of past memories but this morning I spent a few minutes updating the About me … section in this blog. I included the fact that since my early 20’s I started dabbling with starting my own businesses. At the time I got together with a couple friends and opened a liquor store. I was able to secure beer distribution of one of the largest breweries in Peru. The rules are that breweries sell only via distributors and distributors only sell to liquor stores. About a year after, I sold the business due to the fact that I received a scholarship to attend Cornell University and permanently move to the USA.
Last month my wife and I traveled to Portugal for a short break. The person 
sitting next to us in the plane was reading “The Lean Startup” by Eric Ries. Years ago I got a copy and read the book. It contains ideas that not only apply to startups but also to any project in any company regardless of size. The individual mentioned that all employees in his company get a copy of the book when they start and are supposed to read it as part of their employee orientation procedure. I thought that was a good an interesting idea which I have not come across before.
OK, enough chit-chat and let’s get to the subject of this post.
I started work on a project that needs a managed NoSQL database. In the past I have researched Cassandra and MongoDB. As a matter of fact, I am currently working on a different project using MongoDB, but that is a different story. I have read articles about DynamoDB and it seems to be a good fit for a document database for the requirements for the project at hand. In addition, AWS offers the managed aspect. That said; I would like to clarify that MongoDB has an Atlas offering that also runs on AWS. MongoDB is very good but seems to me an overkill (Do not use a cannon to kill a mosquito — Confucius) based on requirements.
It is hard to compare NoSQL databases aside from the type like columnar or document. It seems that they are always adding features to keep and add users. I read the current documentation on DynamoDB and a few articles floating on the web. Most of them compare the initial features of DynamoDB when it was introduced in 2012. For what the service in question requires, probably both DynamoDB and MongoDB would work fine. Due to simplicity of use, performance, and features required, I decided to go with DynamoDB.
I always like to get familiar as much as possible with any tool or component that I will be using in a project. To accomplish this, DynamoDB runs on AWS but one can download a JAR that can be used to experiment and develop software. Once the team is comfortable with the MVP, we can move it to AWS for further development and testing. The beauty is that if one uses the same IAM keys, the development software will run on the cloud with no (perhaps just a few) modifications. In a future post will cover moving the software from development to AWS.
AWS provides extensive documentation of DynamoDB. Not only that, but the JAR can be used on Apple, Linux, and Windows machines. You just need to make sure the software prerequisites are taken care of, install the software, configure it and follow the documentation to interact with DynamoDB using the AWS CLI and the programming language of choice. So far I have taken care of the prerequisites, installation on a Linux machine and interaction with the AWS CLI. In a future post I will describe my experience interacting with it using the JAVA SDK.
I write posts for this blog using MS Word on one of my Windows machines.
For ease of use, I initially decided to test DynamoDB using Windows. I was able to take care of the prerequisites and installation. I was not able to configure the AWS console to interact with DynamoDB. After a few hours I decided to start from scratch on one of my Linux servers. All went well on Linux which will be the platform in which the software will be developed.
Prerequisites
We need to have installed Python 3.3+ in order to be compatible with all AWS services. In order to check the version in my Linux computer I issued the following:
[johncanessa@ceph1 DynamoDB]$ python --version Python 3.6.5 :: Anaconda, Inc. [johncanessa@ceph1 DynamoDB]$
It looks like we have Python version 3.6.5 installed. That is sufficient, but I found out that there is a newer version so why not updated to it. This was accomplished by using the following command:
[johncanessa@ceph1 DynamoDB]$ sudo yum install python36 [sudo] password for johncanessa: Loaded plugins: fastestmirror, langpacks Loading mirror speeds from cached hostfile epel/x86_64/metalink | 17 kB 00:00:00 * base: mirrors.usinternet.com * epel: mirror.steadfastnet.com * extras: ewr.edge.kernel.org * updates: mirrors.usinternet.com epel | 5.3 kB 00:00:00 extras | 3.4 kB 00:00:00 google-chrome | 1.3 kB 00:00:00 sublime-text | 2.9 kB 00:00:00 updates | 3.4 kB 00:00:00 (1/3): epel/x86_64/updateinfo | 977 kB 00:00:00 (2/3): google-chrome/primary | 1.7 kB 00:00:00 (3/3): epel/x86_64/primary_db | 6.8 MB 00:00:00 google-chrome 3/3 Resolving Dependencies --> Running transaction check ---> Package python36.x86_64 0:3.6.8-1.el7 will be installed --> Processing Dependency: python36-libs(x86-64) = 3.6.8-1.el7 for package: python36-3.6.8-1.el7.x86_64 --> Processing Dependency: libpython3.6m.so.1.0()(64bit) for package: python36-3.6.8-1.el7.x86_64 --> Running transaction check ---> Package python36-libs.x86_64 0:3.6.8-1.el7 will be installed --> Finished Dependency Resolution Dependencies Resolved =========================================================================================================================== Package Arch Version Repository Size =========================================================================================================================== Installing: python36 x86_64 3.6.8-1.el7 epel 67 k Installing for dependencies: python36-libs x86_64 3.6.8-1.el7 epel 8.6 M Transaction Summary =========================================================================================================================== Install 1 Package (+1 Dependent package) Total download size: 8.6 M Installed size: 36 M Is this ok [y/d/N]: y Downloading packages: (1/2): python36-3.6.8-1.el7.x86_64.rpm | 67 kB 00:00:00 (2/2): python36-libs-3.6.8-1.el7.x86_64.rpm | 8.6 MB 00:00:00 --------------------------------------------------------------------------------------------------------------------------- Total 6.9 MB/s | 8.6 MB 00:00:01 Running transaction check Running transaction test Transaction test succeeded Running transaction Installing : python36-libs-3.6.8-1.el7.x86_64 1/2 Installing : python36-3.6.8-1.el7.x86_64 2/2 Verifying : python36-3.6.8-1.el7.x86_64 1/2 Verifying : python36-libs-3.6.8-1.el7.x86_64 2/2 Installed: python36.x86_64 0:3.6.8-1.el7 Dependency Installed: python36-libs.x86_64 0:3.6.8-1.el7 Complete! [johncanessa@ceph1 DynamoDB]$
As you can verify in the last few lines of the screen capture, we are now running Python version 3.6.8.
I decided that in order to have additional Python tools I could also install the Python development tools. I did so by issuing the following command:
[johncanessa@ceph1 DynamoDB]$ sudo yum install python36-devel Loaded plugins: fastestmirror, langpacks Loading mirror speeds from cached hostfile * base: mirror.fileplanet.com * epel: muug.ca * extras: repo1.ash.innoscale.net * updates: mirror.fileplanet.com Resolving Dependencies --> Running transaction check ---> Package python36-devel.x86_64 0:3.6.8-1.el7 will be installed --> Processing Dependency: python-rpm-macros for package: python36-devel-3.6.8-1.el7.x86_64 --> Processing Dependency: python3-rpm-macros for package: python36-devel-3.6.8-1.el7.x86_64 --> Running transaction check ---> Package python-rpm-macros.noarch 0:3-24.el7 will be installed --> Processing Dependency: python-srpm-macros for package: python-rpm-macros-3-24.el7.noarch ---> Package python3-rpm-macros.noarch 0:3-24.el7 will be installed --> Running transaction check ---> Package python-srpm-macros.noarch 0:3-24.el7 will be installed --> Finished Dependency Resolution Dependencies Resolved =========================================================================================================================== Package Arch Version Repository Size =========================================================================================================================== Installing: python36-devel x86_64 3.6.8-1.el7 epel 850 k Installing for dependencies: python-rpm-macros noarch 3-24.el7 epel 7.9 k python-srpm-macros noarch 3-24.el7 epel 7.3 k python3-rpm-macros noarch 3-24.el7 epel 6.9 k Transaction Summary =========================================================================================================================== Install 1 Package (+3 Dependent packages) Total download size: 872 k Installed size: 2.6 M Is this ok [y/d/N]: y Downloading packages: (1/4): python-rpm-macros-3-24.el7.noarch.rpm | 7.9 kB 00:00:00 (2/4): python-srpm-macros-3-24.el7.noarch.rpm | 7.3 kB 00:00:00 (3/4): python3-rpm-macros-3-24.el7.noarch.rpm | 6.9 kB 00:00:00 (4/4): python36-devel-3.6.8-1.el7.x86_64.rpm | 850 kB 00:00:00 --------------------------------------------------------------------------------------------------------------------------- Total 1.1 MB/s | 872 kB 00:00:00 Running transaction check Running transaction test Transaction test succeeded Running transaction Installing : python3-rpm-macros-3-24.el7.noarch 1/4 Installing : python-srpm-macros-3-24.el7.noarch 2/4 Installing : python-rpm-macros-3-24.el7.noarch 3/4 Installing : python36-devel-3.6.8-1.el7.x86_64 4/4 Verifying : python36-devel-3.6.8-1.el7.x86_64 1/4 Verifying : python-rpm-macros-3-24.el7.noarch 2/4 Verifying : python-srpm-macros-3-24.el7.noarch 3/4 Verifying : python3-rpm-macros-3-24.el7.noarch 4/4 Installed: python36-devel.x86_64 0:3.6.8-1.el7 Dependency Installed: python-rpm-macros.noarch 0:3-24.el7 python-srpm-macros.noarch 0:3-24.el7 python3-rpm-macros.noarch 0:3-24.el7 Complete! [johncanessa@ceph1 DynamoDB]$
The Amazon documentation makes use of pip3. To make sure we have the latest and grates version of the Python setup tools I used the following:
[johncanessa@ceph1 DynamoDB]$ sudo yum install python36-setuptools Loaded plugins: fastestmirror, langpacks Loading mirror speeds from cached hostfile * base: mirror.fileplanet.com * epel: muug.ca * extras: repo1.ash.innoscale.net * updates: mirror.fileplanet.com Resolving Dependencies --> Running transaction check ---> Package python36-setuptools.noarch 0:39.2.0-3.el7 will be installed --> Finished Dependency Resolution Dependencies Resolved =========================================================================================================================== Package Arch Version Repository Size =========================================================================================================================== Installing: python36-setuptools noarch 39.2.0-3.el7 epel 631 k Transaction Summary =========================================================================================================================== Install 1 Package Total download size: 631 k Installed size: 3.6 M Is this ok [y/d/N]: y Downloading packages: python36-setuptools-39.2.0-3.el7.noarch.rpm | 631 kB 00:00:00 Running transaction check Running transaction test Transaction test succeeded Running transaction Installing : python36-setuptools-39.2.0-3.el7.noarch 1/1 Verifying : python36-setuptools-39.2.0-3.el7.noarch 1/1 Installed: python36-setuptools.noarch 0:39.2.0-3.el7 Complete! [johncanessa@ceph1 DynamoDB]$
Armed with the latest set up tools I installed pip3 as follows:
[johncanessa@ceph1 DynamoDB]$ sudo easy_install-3.6 pip Searching for pip Reading https://pypi.org/simple/pip/ Downloading https://files.pythonhosted.org/packages/5c/e0/be401c003291b56efc55aeba6a80ab790d3d4cece2778288d65323009420/pip-19.1.1-py2.py3-none-any.whl#sha256=993134f0475471b91452ca029d4390dc8f298ac63a712814f101cd1b6db46676 Best match: pip 19.1.1 Processing pip-19.1.1-py2.py3-none-any.whl Installing pip-19.1.1-py2.py3-none-any.whl to /usr/local/lib/python3.6/site-packages Adding pip 19.1.1 to easy-install.pth file Installing pip script to /usr/local/bin Installing pip3 script to /usr/local/bin Installing pip3.7 script to /usr/local/bin Installed /usr/local/lib/python3.6/site-packages/pip-19.1.1-py3.6.egg Processing dependencies for pip Finished processing dependencies for pip [johncanessa@ceph1 DynamoDB]$
We can now verify that all went well with the installation by verifying the current version of pip3 as follows:
[johncanessa@ceph1 DynamoDB]$ pip3 -V pip 19.0.3 from /home/johncanessa/anaconda3/lib/python3.6/site-packages/pip (python 3.6) [johncanessa@ceph1 DynamoDB]$
Installing the AWS CLI
It seems that at this point we have met or exceeded the requirements. We should now be able to install the AWS CLI and then DynamoDB.
To install the AWS CLI using pip3 I issued the following command:
[johncanessa@ceph1 DynamoDB]$ pip3 install awscli --upgrade --user
Collecting awscli
Downloading https://files.pythonhosted.org/packages/0a/f4/15cc8a285f17c8ef031de1ddb01ec5d506a5a104bd4efde77cacc8783df3/awscli-1.16.177-py2.py3-none-any.whl (1.6MB)
100% |████████████████████████████████| 1.6MB 9.4MB/s
Collecting botocore==1.12.167 (from awscli)
Downloading https://files.pythonhosted.org/packages/7b/00/8437c07663969bd219aab33299f17b9d0ecd82622f4e19f482483efbfc6d/botocore-1.12.167-py2.py3-none-any.whl (5.5MB)
100% |████████████████████████████████| 5.5MB 6.5MB/s
Requirement already satisfied, skipping upgrade: docutils>=0.10 in /home/johncanessa/anaconda3/lib/python3.6/site-packages (from awscli) (0.14)
Collecting s3transfer<0.3.0,>=0.2.0 (from awscli)
Downloading https://files.pythonhosted.org/packages/16/8a/1fc3dba0c4923c2a76e1ff0d52b305c44606da63f718d14d3231e21c51b0/s3transfer-0.2.1-py2.py3-none-any.whl (70kB)
100% |████████████████████████████████| 71kB 5.5MB/s
Requirement already satisfied, skipping upgrade: colorama<=0.3.9,>=0.2.5 in /home/johncanessa/anaconda3/lib/python3.6/site-packages (from awscli) (0.3.9)
Collecting rsa<=3.5.0,>=3.1.2 (from awscli)
Downloading https://files.pythonhosted.org/packages/e1/ae/baedc9cb175552e95f3395c43055a6a5e125ae4d48a1d7a924baca83e92e/rsa-3.4.2-py2.py3-none-any.whl (46kB)
100% |████████████████████████████████| 51kB 20.7MB/s
Requirement already satisfied, skipping upgrade: PyYAML<=3.13,>=3.10 in /home/johncanessa/anaconda3/lib/python3.6/site-packages (from awscli) (3.12)
Requirement already satisfied, skipping upgrade: urllib3<1.26,>=1.20; python_version >= "3.4" in /home/johncanessa/anaconda3/lib/python3.6/site-packages (from botocore==1.12.167->awscli) (1.22)
Collecting jmespath<1.0.0,>=0.7.1 (from botocore==1.12.167->awscli)
Downloading https://files.pythonhosted.org/packages/83/94/7179c3832a6d45b266ddb2aac329e101367fbdb11f425f13771d27f225bb/jmespath-0.9.4-py2.py3-none-any.whl
Requirement already satisfied, skipping upgrade: python-dateutil<3.0.0,>=2.1; python_version >= "2.7" in /home/johncanessa/anaconda3/lib/python3.6/site-packages (from botocore==1.12.167->awscli) (2.7.3)
Requirement already satisfied, skipping upgrade: pyasn1>=0.1.3 in /home/johncanessa/anaconda3/lib/python3.6/site-packages (from rsa<=3.5.0,>=3.1.2->awscli) (0.1.9)
Requirement already satisfied, skipping upgrade: six>=1.5 in /home/johncanessa/anaconda3/lib/python3.6/site-packages (from python-dateutil<3.0.0,>=2.1; python_version >= "2.7"->botocore==1.12.167->awscli) (1.11.0)
Installing collected packages: jmespath, botocore, s3transfer, rsa, awscli
Successfully installed awscli-1.16.177 botocore-1.12.167 jmespath-0.9.4 rsa-3.4.2 s3transfer-0.2.1
You are using pip version 19.0.3, however version 19.1.1 is available.
You should consider upgrading via the 'pip install --upgrade pip' command.
[johncanessa@ceph1 DynamoDB]$
Noted that there seems to be a newer version of pip so I decided to upgrade as follows:
[johncanessa@ceph1 DynamoDB]$ pip install --upgrade pip
Collecting pip
Downloading https://files.pythonhosted.org/packages/5c/e0/be401c003291b56efc55aeba6a80ab790d3d4cece2778288d65323009420/pip-19.1.1-py2.py3-none-any.whl (1.4MB)
100% |████████████████████████████████| 1.4MB 9.4MB/s
Installing collected packages: pip
Found existing installation: pip 19.0.3
Uninstalling pip-19.0.3:
Successfully uninstalled pip-19.0.3
Successfully installed pip-19.1.1
[johncanessa@ceph1 DynamoDB]$
In theory all should be well and AWS CLI should have been installed. I should have tested and move on, but I decided to re install the AWS CLI using an installer. This is all documented by Amazon.
The first step is to use cURL to download the AWS installer as illustrated by the following screen capture:
[johncanessa@ceph1 aws-cli]$ curl "https://s3.amazonaws.com/aws-cli/awscli-bundle.zip" -o "awscli-bundle.zip"
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
100 11.2M 100 11.2M 0 0 5065k 0 0:00:02 0:00:02 --:--:-- 5065k
[johncanessa@ceph1 aws-cli]$
In the aws-cli folder that I created before downloading the installer with curl I issued the following command to extract the files:
[johncanessa@ceph1 aws-cli]$ unzip awscli-bundle.zip Archive: awscli-bundle.zip inflating: awscli-bundle/install inflating: awscli-bundle/packages/PyYAML-3.13.tar.gz inflating: awscli-bundle/packages/six-1.12.0.tar.gz inflating: awscli-bundle/packages/ordereddict-1.1.tar.gz inflating: awscli-bundle/packages/urllib3-1.22.tar.gz inflating: awscli-bundle/packages/colorama-0.3.9.tar.gz inflating: awscli-bundle/packages/pyasn1-0.4.5.tar.gz inflating: awscli-bundle/packages/awscli-1.16.177.tar.gz inflating: awscli-bundle/packages/s3transfer-0.2.1.tar.gz inflating: awscli-bundle/packages/urllib3-1.25.3.tar.gz inflating: awscli-bundle/packages/python-dateutil-2.6.1.tar.gz inflating: awscli-bundle/packages/rsa-3.4.2.tar.gz inflating: awscli-bundle/packages/python-dateutil-2.8.0.tar.gz inflating: awscli-bundle/packages/jmespath-0.9.4.tar.gz inflating: awscli-bundle/packages/futures-3.2.0.tar.gz inflating: awscli-bundle/packages/botocore-1.12.167.tar.gz inflating: awscli-bundle/packages/docutils-0.14.tar.gz inflating: awscli-bundle/packages/simplejson-3.3.0.tar.gz inflating: awscli-bundle/packages/virtualenv-15.1.0.tar.gz inflating: awscli-bundle/packages/argparse-1.2.1.tar.gz inflating: awscli-bundle/packages/setup/setuptools_scm-1.15.7.tar.gz [johncanessa@ceph1 aws-cli]$ ls -l total 11564 drwxrwxr-x. 3 johncanessa johncanessa 37 Jun 13 09:37 awscli-bundle -rw-rw-r--. 1 johncanessa johncanessa 11841469 Jun 13 09:36 awscli-bundle.zip [johncanessa@ceph1 aws-cli]$
The final step in the installation of the AWS CLI is to perform the actual installation. I accomplished this by using the following commands:
[johncanessa@ceph1 aws-cli]$ ./awscli-bundle/install -h
Usage: install [options]
Options:
-h, --help show this help message and exit
-i INSTALL_DIR, --install-dir=INSTALL_DIR
The location to install the AWS CLI. The default
value is ~/.local/lib/aws
-b BIN_LOCATION, --bin-location=BIN_LOCATION
If this argument is provided, then a symlink will be
created at this location that points to the aws
executable. This argument is useful if you want to put
the aws executable somewhere already on your path,
e.g. -b /usr/local/bin/aws. This is an optional
argument. If you do not provide this argument you will
have to add INSTALL_DIR/bin to your PATH.
[johncanessa@ceph1 aws-cli]$
[johncanessa@ceph1 aws-cli]$ sudo ./awscli-bundle/install -i /usr/local/aws -b /usr/local/bin/aws
[sudo] password for johncanessa:
Running cmd: /bin/python virtualenv.py --no-download --python /bin/python /usr/local/aws
Running cmd: /usr/local/aws/bin/pip install --no-cache-dir --no-index --find-links file:///home/johncanessa/aws-cli/awscli-bundle/packages/setup setuptools_scm-1.15.7.tar.gz
Running cmd: /usr/local/aws/bin/pip install --no-cache-dir --no-index --find-links file:///home/johncanessa/aws-cli/awscli-bundle/packages awscli-1.16.177.tar.gz
You can now run: /usr/local/bin/aws --version
[johncanessa@ceph1 aws-cli]$
[johncanessa@ceph1 aws-cli]$ aws --version
aws-cli/1.16.177 Python/3.6.5 Linux/3.10.0-957.21.2.el7.x86_64 botocore/1.12.167
The last command shows the version of the AWS CLI installed.
We now need to install the actual DynamoDB software on our Linux computer. Note that we could have skipped this step if we would use the actual service provided by Amazon. As previously mentioned, I always like to experiment and test and at some point I might be incurring charges on my AWS account. For this reason I decided to use a local copy of DynamoDB.
We first need to download DynamoDB. I used the .tar.gz version of the software.
I then proceeded to extract the contents of the archive as illustrated by the following:
[johncanessa@ceph1 DynamoDB]$ gunzip dynamodb_local_latest.tar.gz
Note that I executed the command in the DynamoDB folder which I created for such purpose.
I then expanded the *.tar file as illustrated:
[johncanessa@ceph1 DynamoDB]$ tar -xf dynamodb_local_latest.tar [johncanessa@ceph1 DynamoDB]$ ls -l total 24652 -rw-rw-r--. 1 johncanessa johncanessa 58833 Jun 12 10:20 dynamodb_1.txt -rw-rw-r--. 1 johncanessa johncanessa 1499 Jun 13 08:32 dynamodb_2.txt -rw-r--r--. 1 johncanessa johncanessa 3873611 Feb 7 17:09 DynamoDBLocal.jar <=== -rw-rw-r--. 1 johncanessa johncanessa 21278720 Jun 13 08:15 dynamodb_local_latest.tar drwxr-xr-x. 2 johncanessa johncanessa 4096 Jun 13 08:33 DynamoDBLocal_lib -rw-r--r--. 1 johncanessa johncanessa 8644 Feb 7 17:09 LICENSE.txt -rw-r--r--. 1 johncanessa johncanessa 1128 Feb 7 17:09 README.txt drwxr-xr-x. 2 johncanessa johncanessa 4096 Jun 13 08:33 third_party_licenses
The contents of the DynamoDB folder were displayed.
We are now ready to start the local version of DynamoDB by using the following:
[johncanessa@ceph1 DynamoDB]$ java -Djava.library.path=./DynamoDBLocal_lib -jar DynamoDBLocal.jar Initializing DynamoDB Local with the following configuration: Port: 8000 InMemory: false DbPath: null SharedDb: false shouldDelayTransientStatuses: false CorsParams: *
By default the software listens on port 8000. This can be changed. I left the default port. If you need to use a different port you can get information by using the following command:
$ java -Djava.library.path=./DynamoDBLocal_lib -jar DynamoDBLocal.jar -help
At this point we have DynamoDB running on our Linux computer. Before we can access DynamoDB programmatically or through the AWS Command Line Interface (AWS CLI), we must configure our credentials to enable authorization for our applications. Please note that downloadable DynamoDB requires any credentials to work.
The DynamoDB documentation suggests a possible fake set as follows:
AWS Access Key ID: "fakeMyKeyId" AWS Secret Access Key: "fakeSecretAccessKey"
AWS Credentials
I could have used fake credentials but decided on getting real credentials. You can learn how to get a set of credentials by reading the Amazon documentation. As a hit you must use IAM.
When I generated a set of credentials, I stored them in a *.csv file. The file contains the required credentials plus additional information.
Back at the ranch, I used my real credentials as follows:
[johncanessa@ceph1 .aws]$ aws configure AWS Access Key ID [None]: fakeMyKeyId AWS Secret Access Key [None]: fakeSecretAccessKey Default region name [None]: us-east-1 Default output format [None]: text [johncanessa@ceph1 .aws]$
Note that in order not to disclose my credentials I edited the screen capture and replaced them with fake ones.
To experiment further and verify that all is working as expected I issued the following commands:
[johncanessa@ceph1 aws-cli]$ whoami johncanessa [johncanessa@ceph1 aws-cli]$ aws configure --profile johncanessa AWS Access Key ID [None]: fakeMyKeyId AWS Secret Access Key [None]: fakeSecretAccessKey Default region name [None]: us-east-1 Default output format [None]: text [johncanessa@ceph1 aws-cli]$ [johncanessa@ceph1 .aws]$ aws s3 ls 2018-11-29 12:48:14 deeplens-sagemaker-56f8595a-308c-4b4b-8717-71419afb2dd2 2019-05-14 11:21:26 elasticbeanstalk-us-east-1-807654219084 [johncanessa@ceph1 .aws]$ [johncanessa@ceph1 aws-cli]$ aws s3 ls --profile johncanessa 2018-11-29 12:48:14 deeplens-sagemaker-56f8595a-308c-4b4b-8717-71419afb2dd2 2019-05-14 11:21:26 elasticbeanstalk-us-east-1-807654219084 [johncanessa@ceph1 aws-cli]$
For simplicity I created a default and a johncanessa profiles. I then tested them. They both seem to be working since they returned the same results.
I then went back to the ~/.aws folder and took a look at the files created by the AWS CLI command:
[johncanessa@ceph1 ~]$ cd .aws [johncanessa@ceph1 .aws]$ ls -al total 12 drwxrwxr-x. 2 johncanessa johncanessa 39 Jun 14 09:33 . drwx------. 40 johncanessa johncanessa 4096 Jun 14 09:33 .. -rw-------. 1 johncanessa johncanessa 55 Jun 14 09:33 config -rw-------. 1 johncanessa johncanessa 120 Jun 14 09:33 credentials [johncanessa@ceph1 .aws]$ cat config [profile johncanessa] region = us-east-1 output = text [johncanessa@ceph1 .aws]$ cat credentials [johncanessa] aws_access_key_id = fakeMyKeyId aws_secret_access_key = fakeSecretAccessKey [johncanessa@ceph1 .aws]$
Please note that once again, I edited the screen capture in order not to show my actual credentials.
I also installed the AWS CLI completer. Instructions on how to install it may be found in the AWS web page.
Testing
OK, in theory all should work. We have the local DynamoDB database operational and will give a try creating a database and experimenting with it. All this is suggested and very well described in the DynamoDB instructions provided by Amazon.
We will perform the following set of steps to verify and start to get familiar with the DynamoDB NoSQL database:
- Step 1: Create a Table
- Step 2: Write Data to a Table
- Step 3: Read Data from a Table
- Step 4: Update Data in a Table
- Step 5: Query Data in a Table
- Step 6: Create a Global Secondary Index
- Step 7: Query the Global Secondary Index
- Step 8: Clean Up Resources
Step 1: Create a Table
Before we create a table, let’s see if we are able to list database tables:
[johncanessa@ceph1 ~]$ aws dynamodb list-tables --endpoint-url http://localhost:8000 [johncanessa@ceph1 ~]$
The command did not return errors indicating that we were able to connect to the local instance of the DynamoDB server. As expected we have not created a table so there were no results to return. I tried a different port to verify that the list-tables command would fail.
You can get a list of Amazon DynamoDB command here.
I also tried the following to see if DynamoDB would return some data:
[johncanessa@ceph1 ~]$ aws dynamodb describe-limits --output json --endpoint-url http://localhost:8000
{
"AccountMaxReadCapacityUnits": 80000,
"AccountMaxWriteCapacityUnits": 80000,
"TableMaxReadCapacityUnits": 40000,
"TableMaxWriteCapacityUnits": 40000
}
[johncanessa@ceph1 ~]$
As you can see it did. We are on our way to create a table and experiment with it issuing some commands.
I used the following commands to create the Music table:
[johncanessa@ceph1 ~]$ aws dynamodb create-table \
> --table-name Music \
> --attribute-definitions \
> AttributeName=Artist,AttributeType=S \
> AttributeName=SongTitle,AttributeType=S \
> --key-schema \
> AttributeName=Artist,KeyType=HASH \
> AttributeName=SongTitle,KeyType=RANGE \
> --provisioned-throughput \
> ReadCapacityUnits=10,WriteCapacityUnits=5 \
> --output json --endpoint-url http://localhost:8000
{
"TableDescription": {
"AttributeDefinitions": [
{
"AttributeName": "Artist",
"AttributeType": "S"
},
{
"AttributeName": "SongTitle",
"AttributeType": "S"
}
],
"TableName": "Music",
"KeySchema": [
{
"AttributeName": "Artist",
"KeyType": "HASH"
},
{
"AttributeName": "SongTitle",
"KeyType": "RANGE"
}
],
"TableStatus": "ACTIVE",
"CreationDateTime": 1560608226.787,
"ProvisionedThroughput": {
"LastIncreaseDateTime": 0.0,
"LastDecreaseDateTime": 0.0,
"NumberOfDecreasesToday": 0,
"ReadCapacityUnits": 10,
"WriteCapacityUnits": 5
},
"TableSizeBytes": 0,
"ItemCount": 0,
"TableArn": "arn:aws:dynamodb:ddblocal:000000000000:table/Music",
"BillingModeSummary": {
"BillingMode": "PROVISIONED",
"LastUpdateToPayPerRequestDateTime": 0.0
}
}
}
[johncanessa@ceph1 ~]$
[johncanessa@ceph1 ~]$ aws dynamodb list-tables --output json --endpoint-url http://localhost:8000
{
"TableNames": [
"Music"
]
}
[johncanessa@ceph1 ~]$
The first command creates the table. The second one lists our first table. To get more information on the contents of the create-table command you can to it here. Please remember that I was following instructions on how to get this done on the DynamoDB website.
When DynamoDB has finished creating the Music table, the value of the TableStatus field is set to ACTIVE. You can check this by issuing the following command:
[johncanessa@ceph1 ~]$ aws dynamodb describe-table --table-name Music --output json --endpoint-url http://localhost:8000 | grep TableStatus
"TableStatus": "ACTIVE",
[johncanessa@ceph1 ~]$
Step 2: Write Data to a Table
We will now insert two records into the Music table by using the put-item command.
[johncanessa@ceph1 ~]$ aws dynamodb put-item \
> --table-name Music \
> --item \
> '{"Artist": {"S": "No One You Know"}, "SongTitle": {"S": "Call Me Today"}, "AlbumTitle": {"S": "Somewhat Famous"}, "Awards": {"N": "1"}}' \
> --output json --endpoint-url http://localhost:8000
[johncanessa@ceph1 ~]$
[johncanessa@ceph1 ~]$ aws dynamodb put-item \
> --table-name Music \
> --item \
> '{"Artist": {"S": "No One You Know"}, "SongTitle": {"S": "Call Me Today"}, "AlbumTitle": {"S": "Somewhat Famous"}, "Awards": {"N": "1"}}' \
> --return-values ALL_OLD \
> --output json --endpoint-url http://localhost:8000
{
"Attributes": {
"Artist": {
"S": "No One You Know"
},
"AlbumTitle": {
"S": "Somewhat Famous"
},
"Awards": {
"N": "1"
},
"SongTitle": {
"S": "Call Me Today"
}
}
}
[johncanessa@ceph1 ~]$
On the first command we do not specify the –return-values ALL_OLD argument. We do so in the second insert. Please note that the best this is to read the options and experiment with them.
Step 3: Read Data from a Table
Now we will read one of the items we have inserted into the Music table. We will do so by using the get-item command.
[johncanessa@ceph1 ~]$ aws dynamodb get-item --consistent-read \
> --table-name Music \
> --key '{ "Artist": {"S": "Acme Band"}, "SongTitle": {"S": "Happy Day"}}' \
> --output json --endpoint-url http://localhost:8000
{
"Item": {
"Artist": {
"S": "Acme Band"
},
"AlbumTitle": {
"S": "Songs About Life"
},
"Awards": {
"N": "10"
},
"SongTitle": {
"S": "Happy Day"
}
}
}
[johncanessa@ceph1 ~]$
In the command we specify the “Acme Band” and the song “Happy Day”. The command return such object / document.
Step 4: Update Data in a Table
This example we will update an item in the Music table by issuing the update-item command:
[johncanessa@ceph1 ~]$ aws dynamodb update-item \
> --table-name Music \
> --key '{ "Artist": {"S": "Acme Band"}, "SongTitle": {"S": "Happy Day"}}' \
> --update-expression "SET AlbumTitle = :newval" \
> --expression-attribute-values '{":newval":{"S":"Updated Album Title"}}' \
> --return-values ALL_NEW \
> --output json --endpoint-url http://localhost:8000
{
"Attributes": {
"Artist": {
"S": "Acme Band"
},
"Awards": {
"N": "10"
},
"AlbumTitle": {
"S": "Updated Album Title"
},
"SongTitle": {
"S": "Happy Day"
}
}
}
[johncanessa@ceph1 ~]$
With this command we specify the Band and the SongTitle and update the AlbumTitle. The results are returned to the caller.
Step 5: Query Data in a Table
We can query the Music table by using the query command as illustrated by the following screen capture:
[johncanessa@ceph1 ~]$ aws dynamodb query \
> --table-name Music \
> --key-condition-expression "Artist = :name" \
> --expression-attribute-values '{":name":{"S":"Acme Band"}}' \
> --output json --endpoint-url http://localhost:8000
{
"Items": [
{
"Artist": {
"S": "Acme Band"
},
"Awards": {
"N": "10"
},
"AlbumTitle": {
"S": "Updated Album Title"
},
"SongTitle": {
"S": "Happy Day"
}
}
],
"Count": 1,
"ScannedCount": 1,
"ConsumedCapacity": null
}
[johncanessa@ceph1 ~]$
The command returns the results of the specified query. To get all the records in the Music table we can use the scan command as illustrated:
[johncanessa@ceph1 ~]$ aws dynamodb scan \
> --table-name Music \
> --output json --endpoint-url http://localhost:8000
{
"Items": [
{
"Artist": {
"S": "Acme Band"
},
"Awards": {
"N": "10"
},
"AlbumTitle": {
"S": "Updated Album Title"
},
"SongTitle": {
"S": "Happy Day"
}
},
{
"Artist": {
"S": "No One You Know"
},
"AlbumTitle": {
"S": "Somewhat Famous"
},
"Awards": {
"N": "1"
},
"SongTitle": {
"S": "Call Me Today"
}
}
],
"Count": 2,
"ScannedCount": 2,
"ConsumedCapacity": null
}
[johncanessa@ceph1 ~]$
This last command returns all the records in our Music table.
Step 6: Create a Global Secondary Index
The following AWS CLI example creates a global secondary index AlbumTitle-index for the Music table using the update-table command:
[johncanessa@ceph1 ~]$ aws dynamodb update-table \
> --table-name Music \
> --attribute-definitions AttributeName=AlbumTitle,AttributeType=S \
> --global-secondary-index-updates \
> "[{\"Create\":{\"IndexName\": \"AlbumTitle-index\",\"KeySchema\":[{\"AttributeName\":\"AlbumTitle\",\"KeyType\":\"HASH\"}], \
> \"ProvisionedThroughput\": {\"ReadCapacityUnits\": 10, \"WriteCapacityUnits\": 5},\"Projection\":{\"ProjectionType\":\"ALL\"}}}]" \
> --output json --endpoint-url http://localhost:8000
{
"TableDescription": {
"AttributeDefinitions": [
{
"AttributeName": "Artist",
"AttributeType": "S"
},
{
"AttributeName": "SongTitle",
"AttributeType": "S"
},
{
"AttributeName": "AlbumTitle",
"AttributeType": "S"
}
],
"TableName": "Music",
"KeySchema": [
{
"AttributeName": "Artist",
"KeyType": "HASH"
},
{
"AttributeName": "SongTitle",
"KeyType": "RANGE"
}
],
"TableStatus": "ACTIVE",
"CreationDateTime": 1560608226.787,
"ProvisionedThroughput": {
"LastIncreaseDateTime": 0.0,
"LastDecreaseDateTime": 0.0,
"NumberOfDecreasesToday": 0,
"ReadCapacityUnits": 10,
"WriteCapacityUnits": 5
},
"TableSizeBytes": 146,
"ItemCount": 2,
"TableArn": "arn:aws:dynamodb:ddblocal:000000000000:table/Music",
"BillingModeSummary": {
"BillingMode": "PROVISIONED",
"LastUpdateToPayPerRequestDateTime": 0.0
},
"GlobalSecondaryIndexes": [
{
"IndexName": "AlbumTitle-index",
"KeySchema": [
{
"AttributeName": "AlbumTitle",
"KeyType": "HASH"
}
],
"Projection": {
"ProjectionType": "ALL"
},
"IndexStatus": "CREATING",
"Backfilling": false,
"ProvisionedThroughput": {
"ReadCapacityUnits": 10,
"WriteCapacityUnits": 5
},
"IndexArn": "arn:aws:dynamodb:ddblocal:000000000000:table/Music/index/AlbumTitle-index"
}
]
}
}
[johncanessa@ceph1 ~]$
Step 7: Query the Global Secondary Index
The following AWS CLI example queries a global secondary index AlbumTitle-index on the Music table:
[johncanessa@ceph1 ~]$ aws dynamodb query \
> --table-name Music \
> --index-name AlbumTitle-index \
> --key-condition-expression "AlbumTitle = :name" \
> --expression-attribute-values '{":name":{"S":"Somewhat Famous"}}' \
> --output json --endpoint-url http://localhost:8000
{
"Items": [
{
"Artist": {
"S": "No One You Know"
},
"AlbumTitle": {
"S": "Somewhat Famous"
},
"Awards": {
"N": "1"
},
"SongTitle": {
"S": "Call Me Today"
}
}
],
"Count": 1,
"ScannedCount": 1,
"ConsumedCapacity": null
}
[johncanessa@ceph1 ~]$
Step 8: Clean Up Resources
If you no longer need the Amazon DynamoDB table that we created for the tutorial, you can delete it. This step helps ensure that we are not charged for resources that we are not using. Note that in our local DynamoDB instance, there is no charge for what we keep or not. As an exercise we will delete the Music table using the delete-table command:
[johncanessa@ceph1 ~]$ aws dynamodb delete-table --table-name Music \
> --output json --endpoint-url http://localhost:8000
{
"TableDescription": {
"AttributeDefinitions": [
{
"AttributeName": "Artist",
"AttributeType": "S"
},
{
"AttributeName": "SongTitle",
"AttributeType": "S"
},
{
"AttributeName": "AlbumTitle",
"AttributeType": "S"
}
],
"TableName": "Music",
"KeySchema": [
{
"AttributeName": "Artist",
"KeyType": "HASH"
},
{
"AttributeName": "SongTitle",
"KeyType": "RANGE"
}
],
"TableStatus": "ACTIVE",
"CreationDateTime": 1560608226.787,
"ProvisionedThroughput": {
"LastIncreaseDateTime": 0.0,
"LastDecreaseDateTime": 0.0,
"NumberOfDecreasesToday": 0,
"ReadCapacityUnits": 10,
"WriteCapacityUnits": 5
},
"TableSizeBytes": 146,
"ItemCount": 2,
"TableArn": "arn:aws:dynamodb:ddblocal:000000000000:table/Music",
"BillingModeSummary": {
"BillingMode": "PROVISIONED",
"LastUpdateToPayPerRequestDateTime": 0.0
},
"GlobalSecondaryIndexes": [
{
"IndexName": "AlbumTitle-index",
"KeySchema": [
{
"AttributeName": "AlbumTitle",
"KeyType": "HASH"
}
],
"Projection": {
"ProjectionType": "ALL"
},
"IndexStatus": "ACTIVE",
"ProvisionedThroughput": {
"ReadCapacityUnits": 10,
"WriteCapacityUnits": 5
},
"IndexSizeBytes": 146,
"ItemCount": 2,
"IndexArn": "arn:aws:dynamodb:ddblocal:000000000000:table/Music/index/AlbumTitle-index"
}
]
}
}
[johncanessa@ceph1 ~]$
[johncanessa@ceph1 ~]$ aws dynamodb list-tables --output json --endpoint-url http://localhost:8000 | grep TableStatus
[johncanessa@ceph1 ~]$
After we deleted the table we issued the list-tables command to verify that we deleted our only table and that all is well.
At this point in time there is no code that I will put in my GitHub repository for this post.
In the next post on the DynamoDB I will be experimenting with the JAVA SDK to perform database operations with our local instance.
If you have comments or questions regarding this or any other post in this blog, or if you would like me to help with any phase in the SDLC (Software Development Life Cycle) of a product or service, please do not hesitate and leave me a note below. Requests for help will remain private.
Keep on reading and experimenting. It is the best way to learn and refresh your knowledge!
John
Follow me on Twitter: @john_canessa