SpringBoot -> Kinesis -> S3

Chans·2023년 11월 1일
KinesisSpringbootaws
0

개요

cpc 검증을 하면서 Spring에서 kafka처럼 kinesis가 데이터가 쌓이는지?
그리고 kinesis에 연결된 타겟팅 S3에 데이터가 제대로 적재되는지에 대해 검증하게 되었다.

Data를 Kinesis stream에 넣기

1. dependency

dependencies {
    implementation 'com.amazonaws:amazon-kinesis-client:1.15.0'
  }

2. kinesis client bean 정의

accessKey와 secretKey는 aws계정 key값을 넣어주었다.

@Configuration
public class KinesisConfig {
 
    @Value("${aws.access.key}")
    private String accessKey;
 
    @Value("${aws.secret.key}")
    private String secretKey;
 
    @Bean
    public AmazonKinesis buildAmazonKinesis() {
        BasicAWSCredentials awsCredentials = new BasicAWSCredentials(accessKey, secretKey);
        return AmazonKinesisClientBuilder.standard()
                .withCredentials(new AWSStaticCredentialsProvider(awsCredentials))
                .withRegion(Regions.AP_NORTHEAST_2)
                .build();
    }
}

3. producerService 정의

kinesis 전송을 위한 Service로 전달하려는 객체를 json 타입으로 변경하고 전송한다.

private final ObjectMapper objectMapper;
private final AmazonKinesis amazonKinesis;
 
/**
 * 메시지 전송
 */
public <T> void sendAsyncJsonMessage(String streamName, T message) {
    sendAsync(streamName, message);
}
 
private <T> void sendAsync(String streamName, T message) {
    final String jsonMessage;
    try {
        jsonMessage = objectMapper.writeValueAsString(message);
        PutRecordsRequest putRecordsRequest = new PutRecordsRequest();
        putRecordsRequest.setStreamName(streamName);
        List<PutRecordsRequestEntry> putRecordsRequestEntryList = new ArrayList<>();
        PutRecordsRequestEntry putRecordsRequestEntry = new PutRecordsRequestEntry();
        putRecordsRequestEntry.setData(ByteBuffer.wrap(jsonMessage.getBytes()));
        putRecordsRequestEntry.setPartitionKey(String.format("partitionKey-%d", 0));
        putRecordsRequestEntryList.add(putRecordsRequestEntry);
        putRecordsRequest.setRecords(putRecordsRequestEntryList);
        PutRecordsResult putRecordsResult = amazonKinesis.putRecords(putRecordsRequest);
        log.info("Put Result" + putRecordsResult);
    } catch (JsonProcessingException e) {
        // 실패 로그 쌓기
        String cause = getCause(ExceptionUtils.getRootCause(e));
        log.error(cause);
    }
}
 
private String getCause(Throwable throwable) {
    Throwable rootCause = throwable;
    return rootCause.getClass().getSimpleName() +
           " [" + StringUtils.trimWhitespace(rootCause.getLocalizedMessage()) + "]";
}
  1. kinesis 호출 서비스
    테스트 공고 상세 보기 api 인 '/v1/recruit/view' 호출 시 회원 인증정보인 AlbamonAuthInfoBffRequest 객체값을 전달하도록 테스트 하였다.
@Schema(description = "회원 인증정보")
public class AlbamonAuthInfoBffRequest {
 
    private   String memberId;          //개인회원 ID
    private   String companyId;         //기업회원 ID
    private   String noMemId;           //프랜차이즈회원 ID
 
    private   int franchiseBrandCode;
    private   String franchiseStoreCode;
 
    private   boolean isSignIn;
    private   String memTypeCode;
    private   int appKeyNumber;
    private   int clientType;
    private   int appVersionNumber;
    private   String monguintempscrapfirst;
 
    private boolean   isApp;
    private boolean   isWeb;
    private int appBridgeVersion;
 
    private boolean isSecure;
    private boolean isTestServer;
 
    private String host;
    private boolean isPC;
    private String ipAddress;
    private String referer;
    private String uriPath;
}

RecruitView2Service의 메소드 호출

public RecruitViewDefaultBffResponse getRecruitViewDefaultParam(AlbamonAuthInfoBffRequest authRequest, RecruitViewDefaultBffRequest request) {
        kinesisProducerService.sendAsyncJsonMessage(STREAM_NAME, authRequest);
        return Optional.ofNullable(
                        RecruitViewMapper.INSTANCE.toViewDefaultResponse(client.getRecruitViewDefault(
                                RecruitViewMapper.INSTANCE.toViewDefaultRequest(authRequest, request)).getBody()))
                .orElseThrow();
}
  1. api 호출
    로컬의 Swagger를 통해 api를 호출한다.
  1. 로그 확인
    Application 로그
2023-10-31 15:46:33.896 [DEBUG] [,] [http-nio-30301-exec-10] [Wire.wire] - [0:0:0:0:0:0:0:1][M|GL_1830099] http-outgoing-1 >> "POST / HTTP/1.1[\r][\n]"
2023-10-31 15:46:33.896 [DEBUG] [,] [http-nio-30301-exec-10] [Wire.wire] - [0:0:0:0:0:0:0:1][M|GL_1830099] http-outgoing-1 >> "Host: kinesis.ap-northeast-2.amazonaws.com[\r][\n]"
2023-10-31 15:46:33.896 [DEBUG] [,] [http-nio-30301-exec-10] [Wire.wire] - [0:0:0:0:0:0:0:1][M|GL_1830099] http-outgoing-1 >> "amz-sdk-invocation-id: 1c04da25-b0c2-de16-7d82-87e559c2018b[\r][\n]"
2023-10-31 15:46:33.896 [DEBUG] [,] [http-nio-30301-exec-10] [Wire.wire] - [0:0:0:0:0:0:0:1][M|GL_1830099] http-outgoing-1 >> "amz-sdk-request: ttl=20231031T064723Z;attempt=1;max=4[\r][\n]"
2023-10-31 15:46:33.896 [DEBUG] [,] [http-nio-30301-exec-10] [Wire.wire] - [0:0:0:0:0:0:0:1][M|GL_1830099] http-outgoing-1 >> "amz-sdk-retry: 0/0/500[\r][\n]"
2023-10-31 15:46:33.896 [DEBUG] [,] [http-nio-30301-exec-10] [Wire.wire] - [0:0:0:0:0:0:0:1][M|GL_1830099] http-outgoing-1 >> "Authorization: AWS4-HMAC-SHA256 Credential=AKIASM6O3RUB4KMPDYH3/20231031/ap-northeast-2/kinesis/aws4_request, SignedHeaders=amz-sdk-invocation-id;amz-sdk-request;amz-sdk-retry;content-length;content-type;host;user-agent;x-amz-date;x-amz-target, Signature=1cc40e5c128b814e99f61f1ee697a4b14d8ab46e1bfc4ba9673d65cab1b431c9[\r][\n]"
2023-10-31 15:46:33.896 [DEBUG] [,] [http-nio-30301-exec-10] [Wire.wire] - [0:0:0:0:0:0:0:1][M|GL_1830099] http-outgoing-1 >> "Content-Type: application/x-amz-cbor-1.1[\r][\n]"
2023-10-31 15:46:33.896 [DEBUG] [,] [http-nio-30301-exec-10] [Wire.wire] - [0:0:0:0:0:0:0:1][M|GL_1830099] http-outgoing-1 >> "User-Agent: aws-sdk-java/1.12.408 Mac_OS_X/14.0 OpenJDK_64-Bit_Server_VM/17.0.8+7-b1000.8 java/17.0.8 vendor/JetBrains_s.r.o. cfg/retry-mode/legacy[\r][\n]"
2023-10-31 15:46:33.896 [DEBUG] [,] [http-nio-30301-exec-10] [Wire.wire] - [0:0:0:0:0:0:0:1][M|GL_1830099] http-outgoing-1 >> "X-Amz-Date: 20231031T064633Z[\r][\n]"
2023-10-31 15:46:33.896 [DEBUG] [,] [http-nio-30301-exec-10] [Wire.wire] - [0:0:0:0:0:0:0:1][M|GL_1830099] http-outgoing-1 >> "X-Amz-Target: Kinesis_20131202.PutRecords[\r][\n]"
2023-10-31 15:46:33.896 [DEBUG] [,] [http-nio-30301-exec-10] [Wire.wire] - [0:0:0:0:0:0:0:1][M|GL_1830099] http-outgoing-1 >> "Content-Length: 556[\r][\n]"
2023-10-31 15:46:33.896 [DEBUG] [,] [http-nio-30301-exec-10] [Wire.wire] - [0:0:0:0:0:0:0:1][M|GL_1830099] http-outgoing-1 >> "Connection: Keep-Alive[\r][\n]"
2023-10-31 15:46:33.896 [DEBUG] [,] [http-nio-30301-exec-10] [Wire.wire] - [0:0:0:0:0:0:0:1][M|GL_1830099] http-outgoing-1 >> "[\r][\n]"
2023-10-31 15:46:33.896 [DEBUG] [,] [http-nio-30301-exec-10] [Wire.wire] - [0:0:0:0:0:0:0:1][M|GL_1830099] http-outgoing-1 >> "[0xbf]gRecords[0x9f][0xbf]dDataY[0x1][0xd7]{"memberId":"GL_1830099","companyId":null,"noMemId":null,"franchiseBrandCode":-2147483648,"franchiseStoreCode":"","memTypeCode":"M","appKeyNumber":0,"clientType":14,"appVersionNumber":0,"monguintempscrapfirst":null,"appBridgeVersion":2,"host":"localhost:30301","ipAddress":"0:0:0:0:0:0:0:1","referer":"http://localhost:30301/general-bff/swagger-ui/index.html","uriPath":"/v1/recruit/view","testServer":false,"secure":false,"signIn":true,"app":false,"web":true,"pc":false}lPartitionKeynpartitionKey-0[0xff][0xff]jStreamNamewjmchoi1-clickstream-poc[0xff]"
2023-10-31 15:46:33.923 [DEBUG] [,] [http-nio-30301-exec-10] [Wire.wire] - [0:0:0:0:0:0:0:1][M|GL_1830099] http-outgoing-1 << "HTTP/1.1 200 OK[\r][\n]"
2023-10-31 15:46:33.923 [DEBUG] [,] [http-nio-30301-exec-10] [Wire.wire] - [0:0:0:0:0:0:0:1][M|GL_1830099] http-outgoing-1 << "x-amzn-RequestId: edbf945f-d478-0e1c-b005-31564f420328[\r][\n]"
2023-10-31 15:46:33.923 [DEBUG] [,] [http-nio-30301-exec-10] [Wire.wire] - [0:0:0:0:0:0:0:1][M|GL_1830099] http-outgoing-1 << "x-amz-id-2: hhaF5BtMd0JWVxpp/tKaINuk3WuCGfR/Mvw70j03OZJJiLnUFcXTN4z6q+nvHwEHxarb45MFgwgeglNKnELxPNobC56GixTV[\r][\n]"
2023-10-31 15:46:33.923 [DEBUG] [,] [http-nio-30301-exec-10] [Wire.wire] - [0:0:0:0:0:0:0:1][M|GL_1830099] http-outgoing-1 << "Date: Tue, 31 Oct 2023 06:46:33 GMT[\r][\n]"
2023-10-31 15:46:33.923 [DEBUG] [,] [http-nio-30301-exec-10] [Wire.wire] - [0:0:0:0:0:0:0:1][M|GL_1830099] http-outgoing-1 << "Content-Type: application/x-amz-cbor-1.1[\r][\n]"
2023-10-31 15:46:33.923 [DEBUG] [,] [http-nio-30301-exec-10] [Wire.wire] - [0:0:0:0:0:0:0:1][M|GL_1830099] http-outgoing-1 << "Content-Length: 135[\r][\n]"
2023-10-31 15:46:33.923 [DEBUG] [,] [http-nio-30301-exec-10] [Wire.wire] - [0:0:0:0:0:0:0:1][M|GL_1830099] http-outgoing-1 << "[\r][\n]"
2023-10-31 15:46:33.923 [DEBUG] [,] [http-nio-30301-exec-10] [LoggingManagedHttpClientConnection.onResponseReceived] - [0:0:0:0:0:0:0:1][M|GL_1830099] http-outgoing-1 << HTTP/1.1 200 OK
2023-10-31 15:46:33.923 [DEBUG] [,] [http-nio-30301-exec-10] [LoggingManagedHttpClientConnection.onResponseReceived] - [0:0:0:0:0:0:0:1][M|GL_1830099] http-outgoing-1 << x-amzn-RequestId: edbf945f-d478-0e1c-b005-31564f420328
2023-10-31 15:46:33.923 [DEBUG] [,] [http-nio-30301-exec-10] [LoggingManagedHttpClientConnection.onResponseReceived] - [0:0:0:0:0:0:0:1][M|GL_1830099] http-outgoing-1 << x-amz-id-2: hhaF5BtMd0JWVxpp/tKaINuk3WuCGfR/Mvw70j03OZJJiLnUFcXTN4z6q+nvHwEHxarb45MFgwgeglNKnELxPNobC56GixTV
2023-10-31 15:46:33.923 [DEBUG] [,] [http-nio-30301-exec-10] [LoggingManagedHttpClientConnection.onResponseReceived] - [0:0:0:0:0:0:0:1][M|GL_1830099] http-outgoing-1 << Date: Tue, 31 Oct 2023 06:46:33 GMT
2023-10-31 15:46:33.923 [DEBUG] [,] [http-nio-30301-exec-10] [LoggingManagedHttpClientConnection.onResponseReceived] - [0:0:0:0:0:0:0:1][M|GL_1830099] http-outgoing-1 << Content-Type: application/x-amz-cbor-1.1
2023-10-31 15:46:33.923 [DEBUG] [,] [http-nio-30301-exec-10] [LoggingManagedHttpClientConnection.onResponseReceived] - [0:0:0:0:0:0:0:1][M|GL_1830099] http-outgoing-1 << Content-Length: 135
2023-10-31 15:46:33.923 [DEBUG] [,] [http-nio-30301-exec-10] [MainClientExec.execute] - [0:0:0:0:0:0:0:1][M|GL_1830099] Connection can be kept alive for 60000 MILLISECONDS
2023-10-31 15:46:33.923 [DEBUG] [,] [http-nio-30301-exec-10] [ClockSkewAdjuster.getServerDate] - [0:0:0:0:0:0:0:1][M|GL_1830099] Reported server date (from 'Date' header): Tue, 31 Oct 2023 06:46:33 GMT
2023-10-31 15:46:33.924 [DEBUG] [,] [http-nio-30301-exec-10] [Wire.wire] - [0:0:0:0:0:0:0:1][M|GL_1830099] http-outgoing-1 << "[0xbf]qFailedRecordCount[0x0]gRecords[0x9f][0xbf]nSequenceNumberx849645927032478289648665050013545012414078201711516713010gShardIdtshardId-000000000003[0xff][0xff][0xff]"
2023-10-31 15:46:33.924 [DEBUG] [,] [http-nio-30301-exec-10] [PoolingHttpClientConnectionManager.releaseConnection] - [0:0:0:0:0:0:0:1][M|GL_1830099] Connection [id: 1][route: {s}->https://kinesis.ap-northeast-2.amazonaws.com:443] can be kept alive for 60.0 seconds
2023-10-31 15:46:33.924 [DEBUG] [,] [http-nio-30301-exec-10] [LoggingManagedHttpClientConnection.setSocketTimeout] - [0:0:0:0:0:0:0:1][M|GL_1830099] http-outgoing-1: set socket timeout to 0
2023-10-31 15:46:33.924 [DEBUG] [,] [http-nio-30301-exec-10] [PoolingHttpClientConnectionManager.releaseConnection] - [0:0:0:0:0:0:0:1][M|GL_1830099] Connection released: [id: 1][route: {s}->https://kinesis.ap-northeast-2.amazonaws.com:443][total available: 1; route allocated: 1 of 50; total allocated: 1 of 50]
2023-10-31 15:46:33.924 [DEBUG] [,] [http-nio-30301-exec-10] [AwsResponseHandlerAdapter.handle] - [0:0:0:0:0:0:0:1][M|GL_1830099] Received successful response: 200, AWS Request ID: edbf945f-d478-0e1c-b005-31564f420328
2023-10-31 15:46:33.924 [DEBUG] [,] [http-nio-30301-exec-10] [AwsResponseHandlerAdapter.logHeaderRequestId] - [0:0:0:0:0:0:0:1][M|GL_1830099] x-amzn-RequestId: edbf945f-d478-0e1c-b005-31564f420328
2023-10-31 15:46:33.924 [DEBUG] [,] [http-nio-30301-exec-10] [AwsResponseHandlerAdapter.logExtendedRequestId] - [0:0:0:0:0:0:0:1][M|GL_1830099] AWS Extended Request ID: hhaF5BtMd0JWVxpp/tKaINuk3WuCGfR/Mvw70j03OZJJiLnUFcXTN4z6q+nvHwEHxarb45MFgwgeglNKnELxPNobC56GixTV
2023-10-31 15:46:33.924 [INFO ] [,] [http-nio-30301-exec-10] [KinesisProducerService.sendAsync] - [0:0:0:0:0:0:0:1][M|GL_1830099] Put Result{FailedRecordCount: 0,Records: [{SequenceNumber: 49645927032478289648665050013545012414078201711516713010,ShardId: shardId-000000000003,}],}
2023-10-31 15:46:33.927 [DEBUG] [,] [http-nio-30301-exec-10] [SpringEncoder.logBeforeWrite] - [0:0:0:0:0:0:0:1][M|GL_1830099] Writing [albamon.recruit.info.view.request.RecruitViewDefaultRequest@2f6b2171] using [org.springframework.http.converter.json.MappingJackson2HttpMessageConverter@41fe0ca6]
2023-10-31 15:46:33.927 [DEBUG] [,] [http-nio-30301-exec-10] [LazyTracingFeignClient.execute] - [0:0:0:0:0:0:0:1][M|GL_1830099] Sending a request via tracing feign client [org.springframework.cloud.sleuth.instrument.web.client.feign.TracingFeignClient@437675e1] and the delegate [feign.hc5.ApacheHttp5Client@3b85e54]

S3 데이터

결론

SpringBoot에서 kinesis를 통해 S3로 로그 데이터 적재가 가능함을 확인하였다.

하지만 백엔드를 통해 S3에 데이터를 쌓는게 목적이라면 알바몬에서는 kafka라는 msk를 사용하고 있기에 kafka로 S3 로그 데이터를 쌓아도 된다.

백엔드를 통해 로그를 쌓아야 한다면 kafka와 kinesis의 장단점을 비교해 본 후 사용을 해야될듯 하다.

참고
https://www.baeldung.com/spring-aws-kinesis
https://blog.voidmainvoid.net/297
profile
Chans
이전 포스트

SpringBoot -> Athena -> S3

다음 포스트

Spring Batch에서 AWS Athena Query & Secret Manager

0개의 댓글