String Testing

The use of string tests might not be a common process when testing software; perhaps it should.

So what is String Test?

String Tests are used to test software features when already integrated in the final software and before it is sent to production. When features in the software are being developed, engineers write unit tests to make sure that the different functions and classes work the way they were intended. These tests are typically white box and have reduced scope.

As the software grows and there are multiple dependencies, when changes are made to different parts, it is customary to run redundancy tests at a high level to make sure all is well. String Tests define procedures to check specific feature at a high level. If a test fails then the issue / bug needs to be addressed or the string test needs to be updated.

String Tests should be easy to understand and conduct by the development and QA teams. In general a test written by developers for developers tends to make many assumptions which may or may not be understandable / correct to QA and end users. This is one of the reasons after all tests pass by the development team, then the QA should run the same and additional ones making sure knowledge of the software internals is not needed to run and determine if the tests pass or fail.

The idea when writing a string test is to make it easy for the individual running the test to execute them and verify results. String tests should take a few (5 to 15) minutes on average to execute. The tester should not need a deep understanding of the software system to conduct and most important verify them.

Natural language is not precise. Because of this fact, when writing a sting test the steps and results should be simple to understand, perform and verify. It is said that if you cannot explain something; you do not know it well enough. The development team should be able to write string tests that are simple to understand and replicate by others and the results should always be easy to verify. If they are not, then the software probably has an issue. It is not a good idea to allow testers to interpret the meaning and results of a test. Such practice is a total waste of time and resources.

So how do we write a good string test? Like anything in software, it is a science and an art. Let’s take a stab at writing a good string test. Please note that with time things change. The development team may decide to add (not delete) sections. If that is the case, make sure they are always included, the sections are precise, the process is simple to understand and conduct and the results are easy to verify.

To make sure string test files are easily accessible by testers, write them using a text editor (not a word processor). By doing this the string test files may be copied and displayed on any platform without the need of additional software.

String tests should always have at least the following sections:

Objective
Setup
Stimulus
Results

Each section should be as short as possible but must exactly convey what is expected to be done.

Label each test case to they are easily located within the string test file.

The Objective sections details what the test attempts to prove (or disprove).

The Setup section describes what needs to be done in preparation for the actual test.

The Stimulus section contains what needs to be done with the software to test a single feature. This section may comprise of a single or multiple steps.

The Results section should be simple to describe so the person conducting the test would be confident that the results described match the results obtained. No interpretation should be left to the tester.

Let’s go through writing a string test case for a storage server that I have a very good understanding of. The storage server in question accepts requests to store, query, and retrieve digital objects (e.g., text files, images, videos, etc). In this case we would like to write a string test to verify the operation of “sending a merged bitfile to a storage server”.

A bitfile is an object in the storage server associated with a digital file of any type (i.e., movie, image, word processing document, medical image, etc) in a domain space. For example, when a patient gets an x-ray using one of dozens of different types of equipment (e.g., MRI, CT, etc) the resulting DICOM files representing the images may be sent to a storage server for safe keeping.

The following screen capture shows the contents of this sting test case extracted from a text file (e.g., test_casxfer.st):

************
Test Case 1
************

Objective:
	Verify that a merged bitfile may be sent and properly stored in the storage server.
	A merged bitfile is created by taking a set of files (i.e., text, video, images, etc) and 
	merging them locally (not at the storage server) into a merged bitfile.

	A client is needed to issue the CLI request and 
	a storage server running (in this case) on a Windows machine.

Setup:
	On one or two computers have the client (CLIs) and 
	storage server software properly installed (these items are available in any standard test computer).

	Make sure you use the cashelp.exe CLI to check on other CLIs (e.g., casping.exe, casversion.exe, casquery.exe, etc)
	that may be of use to help determine and possibly address issues when running this test case.

	1.  From a command prompt start the storage server:

		$ sdm -debug

	2. Verify you are using the proper version of the storage software.

		$ casversion ...

	If the expected software version does NOT match; stop and install the proper software version.

	3.  Using a set of 5 to 10 files of any type (files of different types may be mixed together if desired; 
		but is not required) create a local merged bitfile.

		$ casmergefiles ...

Stimulus:
	Send the locally created merged bitfile to the storage server.

	1.  Transfer the merged bitfile to the storage server.

		$ casxfer ...

Results:
	Verify that the merged bitfile is stored in the storage server.

	1. Check that the merged bitfile is in the storage server.

		$ casmergeinfo ...

	2.  Retrieve the merged bitfile from the storage server and extract its contents.

		$ casextractallmerged ...

	Compare the contents of the merged bitfile with the retrieved files. 

	3. List the extracted files (the contents of the merged bitfile were extracted to the specified folder):

		C:\>dir "c:\temp\Retrieve Folder"

		03/17/2018  10:24 AM                15 14feb5f0be5a26e887b05aad2778347a.txt
		03/17/2018  10:24 AM                16 14feb5f0be5a26e887b05aad2778347b.txt
		03/17/2018  10:24 AM                17 14feb5f0be5a26e887b05aad2778347c.txt
		03/17/2018  10:24 AM                31 14feb5f0be5a26e887b05aad2778347d.txt
		03/17/2018  10:24 AM                32 14feb5f0be5a26e887b05aad2778347e.txt
		03/17/2018  10:24 AM                33 14feb5f0be5a26e887b05aad2778347f.txt
		03/17/2018  10:24 AM               192 14feb5f0be5a26e887b05aad27783480.txt
		03/17/2018  10:24 AM               960 14feb5f0be5a26e887b05aad27783481.txt
		03/17/2018  10:24 AM               960 14feb5f0be5a26e887b05aad27783482.txt
		03/17/2018  10:24 AM               960 14feb5f0be5a26e887b05aad27783483.txt
		03/17/2018  10:24 AM             3,298 14feb5f0be5a26e887b05aad27783484.txt
		03/17/2018  10:24 AM                12 14feb5f0be5a26e887b05aad27783485.txt
		03/17/2018  10:24 AM             3,298 14feb5f0be5a26e887b05aad27783486.txt
		03/17/2018  10:24 AM             3,298 14feb5f0be5a26e887b05aad27783487.txt
		03/17/2018  10:24 AM             1,674 14feb5f0be5a26e887b05aad27783488.txt
		03/17/2018  10:24 AM               192 14feb5f0be5a26e887b05aad27783489.txt

	4. List the files that were included when the merged bitfile was generated (in this example the following
	list of files was used):

		C:\>type c:\temp\list_of_text_files.txt

		c:\temp\text files\15_chars.txt
		c:\temp\text files\16_chars.txt
		c:\temp\text files\17_chars.txt
		c:\temp\text files\31_chars.txt
		c:\temp\text files\32_chars.txt
		c:\temp\text files\33_chars.txt
		c:\temp\text files\ach.txt
		c:\temp\text files\ach_1.txt
		c:\temp\text files\ach_2.txt
		c:\temp\text files\ach_3.txt
		c:\temp\text files\guid_list.txt
		c:\temp\text files\hello_world.txt
		c:\temp\text files\list.txt
		c:\temp\text files\my_guid_list.txt
		c:\temp\text files\query_times.txt
		c:\temp\text files\retrieved.txt

	5. Verify that the list of files matches the available files (in this case the following folder
	was used):

		C:\>dir "c:\temp\text files"

		06/24/2016  09:21 AM                15 15_chars.txt
		06/24/2016  09:22 AM                16 16_chars.txt
		06/24/2016  09:22 AM                17 17_chars.txt
		06/24/2016  09:23 AM                31 31_chars.txt
		06/24/2016  09:23 AM                32 32_chars.txt
		06/24/2016  09:24 AM                33 33_chars.txt
		02/09/2016  04:03 PM               192 ach.txt
		02/09/2016  02:54 PM               960 ach_1.txt
		02/09/2016  02:58 PM               960 ach_2.txt
		02/09/2016  03:38 PM               960 ach_3.txt
		04/12/2016  09:16 AM             3,298 guid_list.txt
		06/24/2016  08:15 AM                12 hello_world.txt
		04/07/2016  05:02 PM             3,298 list.txt
		05/04/2016  09:45 AM             3,298 my_guid_list.txt
		04/05/2016  01:17 PM             1,674 query_times.txt
		02/29/2016  06:57 PM               192 retrieved.txt

	6. As an added bonus one may select a random number of the files and compare their contents.
	They should be identical down to the byte count and contents.

Along the way, it was mentioned that to get help with the storage server CLIs one may use cashelp.exe to get the specifics on their use. Following is a screen dump of the cashelp.exe CLI:

C:\>cashelp

        casappendtrailer        APPEND trailers to bitfiles in a folder.
        cascache                Display the STATE of all configured disk caches.
        cascachestostore        Remote disk cache(s) that may be used to store bitfile(s) for specified group ID.
        caschangegroup          Issue a request to a iCAS server to CHANGE the group of the specified GUID.

        cascheckbitfiles        CHECK a bitfile or set of bitfiles specified by GUID in a specified iCAS server.
        cascheckcache           Check the contents of a disk cache.
        cascheckcifs            Check the contents of the CIFS_TBL table against the specified file system.
        cascheckdigest          Check if the specified digest matches the file and a GUID in a iCAS server.

        caschecklic             Check if the specified license is valid.
        cascopymissing          Copy damaged \ missing bitfiles to iCAS from off-line disk cache.
        cascountbitfiles        Returns the count of bitfiles in all the folders of all disk caches.
        casdelete               Delete a GUID or set of GUIDs from the specified iCAS server.

        casdumpmerged           Dump the contents of a LOCAL merged bitfile.
        casexec                 Execute a command on the host of a iCAS server.
        casexercise             Store, query, retrieve and delete a test file to / from a iCAS server.
        casextractallmerged     Extract from a MERGED bitfile the files to the specified path.

        casfindissues           Check if bitfiles exist in configured disk caches and if they are damaged.
        casgendigest            Compute the MD5 digest for the specified file.
        casgetevents            Register to receive iCAS server notification events.
        casgetguids             Return all GUIDs for a specified date and group (also see caslistguids).

        casguidsfordate         Return all GUIDs for a specified date.
        casguidstopaths         Return the full paths to specified bitfiles in disk caches.
        cashelp                 List all the CLI commands with a brief description.
        caslistguids            Return all GUIDs for a specified date and group (also see casgetguids).

        caslistmissing          Check all the bitfiles for the specified date generating a list of missing ones.
        casloadbalance          Return load balancing information on a iCAS server cluster.
        caslogoff               Log OFF a user to a specified iCAS server.
        caslogon                Log ON a user from a specified iCAS server.

        casmerge                MERGE a set of bitfiles into a single bitfile.
        casmergefiles           MERGE a set of files into a MERGED bitfile without a iCAS server.
        casmergeinfo            Provide information about a MERGED bitfile.
        casmigrate              Remigrate all bitfiles from a iCAS server based on migration configuration.

        casmvbfs                Move all bitfiles from a source to a destination disk cache.
        casmsmq                 Test the operation of iCAS server events using the MSSQ.
        casparsesyngoblob       Extract from a Siemens Syngo blob all DICOM objects.
        casping                 Verify network connectivity to a specified iCAS server.

        casprefetch             Prefetch a bitfile from a a remote iCAS server.
        caspull                 Pull missing bitfiles from remote into local iCAS (needs configured PULL group).
        caspurge                Purge from the disk cache(s) temporary or migrated bitfiles.
        caspush                 Push into an iCAS a file/bitfile skipping a head and/or a tail.

        casquery                Issue a query for the specified GUID to the specified iCAS server.
        casreadtrailer          Read the trailer from the specified bitfile.
        casremovetrailer        Remove bitfile trailers from specified bitfiles.
        casretrieve             Issue a retrieve request for the specified GUID to the specified iCAS server.

        casretrievebyoffset     Retrieve a single bitfile from a MERGED bitfile.
        casroute                Route a set of one (1) or more bitfiles from one to another iCAS server.
        casshutdown             Issue a shutdown request to the specified iCAS server.
        cassnmp                 Request information via SNMP about system services running on a computer.

        cassplit                SPLIT a merged bitfile into its bitfiles.
        casstore                STORE to the specified iCAS server the specified file.
        cassynch                SYNCHRONIZE the contents of the specified iCAS servers.
        cassyngoimages          EXTRACT a set of DICOM objects from a bitfile stored by a Siemens Syngo PACS.

        castestio               TEST I/O operations on disk caches.
        castime                 TIME transfer operations to an iCAS.
        castouch                TOUCH the specified set of files.
        casversion              Retrieve software VERSION information from the specified iCAS server.

        casxfer                 TRANSFER a single bitfile from a client to an iCAS server.

        To obtain ADDITIONAL information for each command line interface (CLI); from a command prompt
        invoke the command using the "-h" help flag:

        e.g.,   casxfer -h

How do we determine if the storage server is up and running? Using the cashelp.exe CLI we can easily determine that we could use casping.exe:

C:\>casping
SockToSendReceive <<< connect WSAECONNREFUSED remoteIP ==>192.168.1.110<== localIP ==>192.168.1.110<== port: 4444 line: 2404 file ==>c:\sencorsource\source\sock.c<==
SockSendCommand <<< SockToSendReceive WAR_CONNECTION_REFUSED serverIP ==>192.168.1.110<== serverPort: 4444 requestType: 0x00000001 line: 3715 file ==>c:\sencorsource\source\sock.c<==
CASPing <<< SockSendCommand WAR_CONNECTION_REFUSED line: 16536 file ==>c:\sdm\source\sdmapi.c<==
casping <<< CASPing WAR_CONNECTION_REFUSED serverIP ==>192.168.1.110<== serverPort: 4444 line: 524
casping <<< retVal: -1002 line: 629 file ==>c:\sdm\source\casping.c<==

It seems that the storage server is not running so we probably need to start it. Before we do we could see is there is something else we could get out of the casping.exe CLI by issuing:

C:\>casping -h

        casping [iCAS TCP/IP]

        This command issues a "ping" to the specified iCAS.
        To determine connectivity to a storage server one should first try the TCP/IP ping utility.
        This may or may not work if the computer hosting the storage server has "ping" disabled.
        To determine connectivity at the storage server (application) level one should use the casping CLI.

        -a <ACCESS code>
        -d <DELAY in seconds>
        -h display this HELP screen
        -i send an ICMP packet to the computer hosting the storage server

        -p <storage server PORT>
        -s <storage server TCP/IP>
        -t TESTING ONLY - casping CLI hungs
        -u <USER name>

        -v run this command in VERBOSE mode

This command issues a “ping” to the specified iCAS.

To determine connectivity to a storage server one should first try the TCP/IP ping utility.

This may or may not work if the computer hosting the storage server has “ping” disabled.

To determine connectivity at the storage server (application) level one should use the casping CLI.

Now let’s determine how to start the storage server:

C:\>sdm -h
SDM -install          to install the service
SDM -remove           to remove the service
SDM -debug <-recover> to run as a console app for debugging

StartServiceCtrlDispatcher being called.
This may take several seconds.  Please wait.

Looks good so let’s do it:

C:\>sdm -debug
CmdDebugService <<< Debugging SDM !!!
SDM <<< starting line: 314 ...
SDM <<< _WIN32 defined line: 1079
SDM <<< _WIN64 NOT defined line: 1087
:::: :::: :::: 
SDM <<< enableCheckGroupTable: 0 (bool) controlled by EnableSDMCheckGroupTbl line: 5506
SDM <<< pollForRemoteGUIDs: 0 (bool) controlled by PollForRemoteGUIDs line: 5623
SDM <<< casXferEncrypted: 0 (bool) controlled by CASXferEncrypted line: 5742
SDM <<< !!!! The SDM service; part of the iCAS; is READY to process requests !!!! line: 5756
SDM <<< LISTENING for requests line: 5879 ...
:::: :::: ::::

Please note that for this test we wish to start the storage server (sdm.exe) interactively. The storage server can also be started as a service but all messages would be written to a log file. It is easier to run it interactively for debugging purposes.

After the storage server is started we could use the casping.exe CLI:

C:\>casping
casping <<< iCAS reply from 192.168.1.110

Or:

C:\>casping -i
casping <<< ICMP reply from 192.168.1.110
casping <<< iCAS reply from 192.168.1.110

Seems like we get a reply. All is well.

When testers are about to create a merged bitfile; help is always at their fingertips:

C:\>casmergefiles -h

        casmergefiles   file_1 ... file_n -m <full_path_for_merged_bitfile>

        This command LOCALLY (no need to access an iCAS server instance) merges into
        a merged bitfile a set of one (1) or more files (NOT bitfiles) specified by
        full path name.
        This CLI returns the GUID associated with the newly created merged bitfile.
        It is up to the caller to transfer the generated merged bitfile to an instance
        of an iCAS server using the casxfer.exe CLI.

        MERGE_MIN_GUID_COUNT: 1 MERGE_MAX_GUID_COUNT: 32768

        file_i  A file to be merged

        -d DELETE the original bitfiles that were merged
        -f full path to a text FILE holding a list of bitfiles to merge
           (you may use two '-' characters to comment out entries)
        -g <merged file GROUP ID>
        -h display this HELP screen

        -l full path to a text file in which the LIST of GUIDs and merged GUID
           (last entry) are returned
        -m <merged bitfile PATH>
        -r <REPOSITORY full path>
        -v run the command in VERBOSE mode

The same holds true when atester is ready to use the casxfer.exe CLI:

C:\>casxfer -h

        casxfer bitfile

        Transfer (NOT store) a single or multiple bitfile(s) (NOT file(s)) from a
        client to an iCAS instance.
        This CLI is typically used to transfer from a client a newly created
        merged bitfile (using the CLI casmergefiles.exe) to an iCAS server.

        bitfile         FULL PATH to a single bitfile to be transfered to an iCAS

        -a <ACCESS code>
        -c <path to a disk CACHE folder>
        -d <DELAY in seconds>
        -f <FILE holding a list of full paths of bitfiles to transfer>

        -h display this HELP screen
        -p <iCAS PORT>
        -r REMOVE bitfile in iCAS before sending (ignores returned status)
        -s <iCAS TCP/IP>

        -u <USER name>
        -v run the command in VERBOSE mode

And also when checking the results of the test:

C:\>casmergeinfo -h

        casmergeinfo merged_GUID

        Issue a query for the specified merged bitfile (by GUID)
        to the specified storage server.

        merged_GUID     GUID of a merged bitfile

        -a <ACCESS code>
        -g <GUID of interest in the merged bitfile>
        -f <FILE in which to write the list of GUIDs>
        -h display this HELP screen

        -p <storage server PORT>
        -s <storage server TCP/IP>
        -u <USER name>
        -v run the command in VERBOSE

As you can see, a person with almost no experience in the architecture, design or implementation of the storage server and its API is able to generate off-line a merged bitfile, send it to an instance of the storage server and then verify that the contents of the stored merged bitfile match what were specified.

Always keep in mind that natural language is not exact and precise. Repeating the same thing over and over never helps. You need to express the ideas using different words. If you cannot describe something; you do not understand it. Simple tools (e.g., text editor) can be successfully used in software development. You do not need expensive tools to produce high quality software. Always keep in mind the KISS (Keep it Simple and Short) rule.

If you have comments or questions regarding this or any other post in this blog, please leave me a comment. I will respond to it as soon as possible.

Happy software development;

John

john.canessa@gmail.com

Follow me on Twitter:  @john_canessa

Leave a Reply

Your email address will not be published. Required fields are marked *

This site uses Akismet to reduce spam. Learn how your comment data is processed.