Scribd
Upload
26 views3 pages

DynamoDB LogstashPipelines Document

technical

Uploaded by

anil85
Copyright
© © All Rights Reserved
We take content rights seriously. If you suspect this is your content, claim it here.
Available Formats
Download as DOCX, PDF, TXT or read online on Scribd
26 views3 pages

DynamoDB LogstashPipelines Document

technical

Uploaded by

anil85
Copyright
© © All Rights Reserved
We take content rights seriously. If you suspect this is your content, claim it here.
Available Formats
Download as DOCX, PDF, TXT or read online on Scribd
You are on page 1/ 3

Overview

 Create a new branch (feat_lgp-ddb-repo is a good name)


 Implement the DynamoDB retrieval methods
in LogstashPipelineDynamoDBRepository.mts using the skeleton provided.
 Before committing changes, ensure the code format is standardized
using prettier: npm run format
 Validate logic by running the unit tests: npm run test-packages
 Open a PR to make any comments on the code, even if the PR stays in draft.
Avoid commenting directly on commits because it's difficult to find those
comments later.
Architecture
DynamoDB
We're using a single-table design in DynamoDB, which means we're using
complicated Partition Key pk / Sort Keys sk to let everything reside in the same
place.

Ke
Value
y

DATA#logstash-
pk
pipeline

sk CURRENT#(id)

HISTORY#(id)#(vers
sk
ion)

Your eyes do not deceive you -- there's two sk listed. The sk:CURRENT#(id) is the
one that's used in all the operations. BUT we're also keeping history for these items,
so every time we change the current one we're also adding a history entry.
 id: The unique ID generated by LogstashPipelineDynamoDBMapper
 version: The timestamp the change was made, e.g. 1727251684123
AWS SDK
Use the AWS SDK v3. The imports etc are already present in the file. Use
the DocumentClient in favor of the standard Client as it does un/marshalling for you.
Data Output
The API uses the ts-results-es package to return the appropriate data, or an error
object. (e.g. Result<TData, TError>). There are three helper types that clarify what
format the returned data should be:

Implementati
Type Example Use Case Purpose
on

Return a single
item from a single
invocation, or
error.
SingleResult Result<T,
getItemById(id) Error could be
<T> ErrorType>
database
misconfiguration
or item not found,
for example

Return multiple
items from a
single invocation,
or error.
Error could be
MultiResult< Result<T[],
getAllItems() database
T> ErrorType>
misconfiguration,
for example (no
items found can
just be an empty
array)

Return a list of
single items from
multiple
invocations, or
`Result<Result
BatchResult< saveMultipleItems(ite error -- or an error
<T, ErrorType>,
T> ms) if something
ErrorType>
prevented any of
the invocations
(e.g. database
misconfiguration)

deleteItems(items): BatchResult
 Take an array of items and delete them from the database - DeleteCommand
o Don't delete history

 If none of the deletions can be invoked at all, return an ErrorType.


 If the deletions can be invoked, then return a list of those deletions or error.
findItems(filter, sort, offset, limit): MultiResult
The "filter" and "sort" part of this is still being designed
 For now, treat this like getAll(offset, limit) and we'll implement filtering and
sorting later - this should be possible with QueryCommand
o Remember to query with the specific partition key, which should be the
same for all Logstash Pipeline items.
 Return an array of results.
o Not the raw responses from DynamoDB, just the items themselves.

getItems(ids): BatchResult
 Get each item by its ID - GetCommand (but see below)
 If the lookup(s) can't be invoked at all, return an ErrorType.
 If the lookup(s) can be invoked, return a list of those items or error (for
example, if not found).
There's the possibility that DynamoDB will allow you to do all of this at once
(BatchGetItem, I think) instead of making multiple GetItem calls.
saveItems(items): BatchResult
 Save each item - PutCommand
 If the writes can't be invoked at all, return an ErrorType.
 If the write can be invoked, return a list of those items or error (for example,
if SDK error).
There's the possibility that DynamoDB will allow you to do all of this at once
(BatchPutItem, I think) instead of making multiple GetItem calls.
Additional Notes
 I plan to update the unit tests as we go; right now they're pretty sparse.
 Let me know if you have any questions.

You might also like