Amazon AWS, DynamoDB, Java

How to fetch all items from a DynamoDB table in Java?

Recently, I wanted to fetch all the items from a DynamoDB table using Scan operation, and it turned out to be more trickier than I originally thought. This is because of the fact that Scan operation has a data size limit of 1 MB. That is, if the total number of scanned items exceeds the maximum data size limit of 1 MB, the scan stops and it does not return all the records. The Query operator has 1 MB limit as well. If you want to fetch less than 1 MB data, you don’t have to worry about anything. However, if you want to fetch data exceeding 1 MB limit then you will have to use LastEvaluatedKey & ExclusiveStartKey.

As per Amazon documentation:

The data returned from a Query or Scan operation is limited to 1 MB; this means that if you scan a table that has more than 1 MB of data, you’ll need to perform another Scan operation to continue to the next 1 MB of data in the table. If you query for specific attributes that match values that amount to more than 1 MB of data, you’ll need to perform another Query request for the next 1 MB of data. The second query request uses a starting point (ExclusiveStartKey) based on the key of the last returned value (LastEvaluatedKey) so you can progressively query or scan for new data in 1 MB increments. The LastEvaluatedKey is null when the entire Query or Scan result set is complete (i.e. the operation processed the “last page”).

Here is a sample program to fetch all the records from a DynamoDB table in Java:

	private static ArrayList<Long> fetchItems() {
		
		ArrayList<Long> ids = new ArrayList<Long>();

		ScanResult result = null;

		do{
			ScanRequest req = new ScanRequest();
			req.setTableName(tableName);

			if(result != null){
				req.setExclusiveStartKey(result.getLastEvaluatedKey());
			}
			
			result = client.scan(req);

			List<Map<String, AttributeValue>> rows = result.getItems();

			for(Map<String, AttributeValue> map : rows){
				try{
					AttributeValue v = map.get("STUDENT_ID");
					String id = v.getS();
					ids.add(Long.parseLong(id));
				} catch (NumberFormatException e){
					System.out.println(e.getMessage());
				}
			}
		} while(result.getLastEvaluatedKey() != null);
		
		System.out.println("Result size: " + ids.size());

		return ids;
	}
Advertisements

One thought on “How to fetch all items from a DynamoDB table in Java?

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s