Amazon AWS, DynamoDB, Java

How to fetch all items from a DynamoDB table in Java?

Recently, I wanted to fetch all the items from a DynamoDB table using Scan operation, and it turned out to be more trickier than I originally thought. This is because of the fact that Scan operation has a data size limit of 1 MB. That is, if the total number of scanned items exceeds the maximum data size limit of 1 MB, the scan stops and it does not return all the records. The Query operator has 1 MB limit as well. If you want to fetch less than 1 MB data, you don’t have to worry about anything. However, if you want to fetch data exceeding 1 MB limit then you will have to use LastEvaluatedKey & ExclusiveStartKey.

As per Amazon documentation:

The data returned from a Query or Scan operation is limited to 1 MB; this means that if you scan a table that has more than 1 MB of data, you’ll need to perform another Scan operation to continue to the next 1 MB of data in the table. If you query for specific attributes that match values that amount to more than 1 MB of data, you’ll need to perform another Query request for the next 1 MB of data. The second query request uses a starting point (ExclusiveStartKey) based on the key of the last returned value (LastEvaluatedKey) so you can progressively query or scan for new data in 1 MB increments. The LastEvaluatedKey is null when the entire Query or Scan result set is complete (i.e. the operation processed the “last page”).

Here is a sample program to fetch all the records from a DynamoDB table in Java:

	private static ArrayList<Long> fetchItems() {
		
		ArrayList<Long> ids = new ArrayList<Long>();

		ScanResult result = null;

		do{
			ScanRequest req = new ScanRequest();
			req.setTableName(tableName);

			if(result != null){
				req.setExclusiveStartKey(result.getLastEvaluatedKey());
			}
			
			result = client.scan(req);

			List<Map<String, AttributeValue>> rows = result.getItems();

			for(Map<String, AttributeValue> map : rows){
				try{
					AttributeValue v = map.get("STUDENT_ID");
					String id = v.getS();
					ids.add(Long.parseLong(id));
				} catch (NumberFormatException e){
					System.out.println(e.getMessage());
				}
			}
		} while(result.getLastEvaluatedKey() != null);
		
		System.out.println("Result size: " + ids.size());

		return ids;
	}
Amazon AWS, DynamoDB

java.lang.IllegalStateException: Connection not obtained from this manager

Got this error today morning while making a call to DynamoDB from a GWT Maven application. This call normally has no issues. At first, I thought it might have to do with my credentials (like I might have entered my access/secret keys wrong). However, digging a little bit into it, I found that it has to do with Maven dependency in POM.xml.

Basically, I was not using the latest version of the Amazon AWS as the dependency in my POM.xml. I changed the below dependency:

		<dependency>
			<groupId>com.amazonaws</groupId>
			<artifactId>aws-java-sdk</artifactId>
			<version>1.3.0</version>
		</dependency>

to

		<dependency>
			<groupId>com.amazonaws</groupId>
			<artifactId>aws-java-sdk</artifactId>
			<version>1.3.10</version>
		</dependency>

It’s just a matter of time before this version (1.3.10) becomes obsolete too. Therefore, make sure that you always have the latest version in your pom.xml file.

This fix works for me. Hope it works for you too.