1년 지난 데이터를 지우는 작업을 하다가 테스트 데이터를 만들기 위해
CompletableFuture
를 통해 비동기로 데이터를 Insert 하는 테스트 케이스를 만들어
진행하던 중에 궁굼한 점(? 의문점 ?)이 생겨 기록해보려고 한다.
static List<List<Jobs>> jobsList;
static int resultSize;
static final int JOBS_SIZE = 3_000;
static final int BATCH_SIZE = 50;
@BeforeEach
public void beforeInit() {
jobsList = new ArrayList<>();
resultSize = 0;
List<Jobs> list = new ArrayList<>();
for (int i = 1; i <= JOBS_SIZE; i++) {
Jobs jobs = Jobs.builder()
.seq(i)
.name(String.valueOf(i))
.build();
list.add(jobs);
if (i % BATCH_SIZE == 0 || i == JOBS_SIZE) {
jobsList.add(list);
list = new ArrayList<>();
}
}
}
private CompletableFuture<Void> future(List<Jobs> list, Executor executor) {
log.info("future list.size: {}", list.size());
return CompletableFuture
.supplyAsync(() -> repository.saveAll(list), executor)
.thenAccept(future -> {
synchronized (this) {
resultSize += future.size();
log.info("Save Completed... list.size: {}, completedCount: {}, ({}/{})", BATCH_SIZE, resultSize, (resultSize / BATCH_SIZE), (JOBS_SIZE / BATCH_SIZE));
}
});
}
<Completable>
를 먼저 만들기 @Test
@DisplayName("Test 1")
public void test1() throws Exception{
log.info("Thread Pool Size: {}", threadPool);
log.info("Futures start batch Size: {}", BATCH_SIZE);
Executor executor = Executors.newFixedThreadPool(threadPool);
LocalDateTime before = DateUtils.now();
List<CompletableFuture<Void>> futures = jobsList.stream()
.map(list -> future(list, executor))
.collect(Collectors.toList());
List<Void> result = futures.stream().map(CompletableFuture::join).collect(Collectors.toList());
LocalDateTime after = DateUtils.now();
log.info("Insert running time: {}s", Duration.between(before, after).getSeconds());
log.info("List Create Completed Size: {}", resultSize);
log.info("3s After Delete Run...");
Thread.sleep(3000L);
}
2023-03-29 12:06:56,178 [INFO ] com.test.InsertSpeedTest.future:91 - future list.size: 50
2023-03-29 12:06:56,179 [INFO ] com.test.InsertSpeedTest.future:91 - future list.size: 50
2023-03-29 12:06:56,179 [INFO ] com.test.InsertSpeedTest.future:91 - future list.size: 50
2023-03-29 12:06:56,180 [INFO ] com.test.InsertSpeedTest.future:91 - future list.size: 50
2023-03-29 12:06:56,180 [INFO ] com.test.InsertSpeedTest.future:91 - future list.size: 50
// ... 같은 로드 총 60개
2023-03-29 11:23:17,970 [INFO ] com.test.InsertSpeedTest.lambda$test1$2:102 - Save Completed... list.size: 50, completedCount: 50, (1/60)
2023-03-29 11:23:17,970 [INFO ] com.test.InsertSpeedTest.lambda$test1$2:102 - Save Completed... list.size: 50, completedCount: 100, (2/60)
2023-03-29 11:23:17,971 [INFO ] com.test.InsertSpeedTest.lambda$test1$2:102 - Save Completed... list.size: 50, completedCount: 150, (3/60)
2023-03-29 11:23:17,971 [INFO ] com.test.InsertSpeedTest.lambda$test1$2:102 - Save Completed... list.size: 50, completedCount: 200, (4/60)
2023-03-29 12:07:12,975 [INFO ] com.test.InsertSpeedTest.lambda$future$1:96 - Save Completed... list.size: 50, completedCount: 250, (5/60)
2023-03-29 12:07:12,975 [INFO ] com.test.InsertSpeedTest.lambda$future$1:96 - Save Completed... list.size: 50, completedCount: 300, (6/60)
2023-03-29 12:07:12,975 [INFO ] com.test.InsertSpeedTest.lambda$future$1:96 - Save Completed... list.size: 50, completedCount: 350, (7/60)
2023-03-29 12:07:12,975 [INFO ] com.test.InsertSpeedTest.lambda$future$1:96 - Save Completed... list.size: 50, completedCount: 400, (8/60)
future(list, executor)
비동기 함수를 수행 후 그 다음 join
메소드를 통해 전부 Blocking을 진행한다. @Test
@DisplayName("Test 2")
public void test2() throws Exception{
log.info("Thread Pool Size: {}", threadPool);
log.info("Futures start");
Executor executor = Executors.newFixedThreadPool(threadPool);
LocalDateTime before = DateUtils.now();
jobsList.stream()
.map(list -> future(list, executor))
.map(CompletableFuture::join)
.collect(Collectors.toList());
LocalDateTime after = DateUtils.now();
log.info("Insert running time: {}s", Duration.between(before, after).getSeconds());
log.info("List Create Completed Size: {}", resultSize);
log.info("3s After Delete Run...");
Thread.sleep(3000L);
}
2023-03-29 12:07:16,668 [INFO ] com.test.InsertSpeedTest.future:91 - future list.size: 50
2023-03-29 12:07:17,504 [INFO ] com.test.InsertSpeedTest.lambda$future$1:96 - Save Completed... list.size: 50, completedCount: 50, (1/60)
2023-03-29 12:07:17,504 [INFO ] com.test.InsertSpeedTest.future:91 - future list.size: 50
2023-03-29 12:07:17,927 [INFO ] com.test.InsertSpeedTest.lambda$future$1:96 - Save Completed... list.size: 50, completedCount: 100, (2/60)
Completable::join
수행시 하나씩 직렬적으로 수행되는 것을 볼 수 있다. 이 것은 하나의 맵에서 직렬로 코드가 수행되기 때문이다. @Test
@DisplayName("Test 3")
public void test3() throws Exception{
log.info("Thread Pool Size: {}", threadPool);
log.info("Futures start");
Executor executor = Executors.newFixedThreadPool(threadPool);
LocalDateTime before = DateUtils.now();
jobsList.stream()
.map(list -> future(list, executor))
.map(CompletableFuture::join)
.parallel()
.collect(Collectors.toList());
LocalDateTime after = DateUtils.now();
log.info("Insert running time: {}s", Duration.between(before, after).getSeconds());
log.info("List Create Completed Size: {}", resultSize);
log.info("3s After Delete Run...");
Thread.sleep(3000L);
}
전체 로그 접기/펼치기 (Velog 안됨)
2023-03-29 12:25:00,245 [INFO ] com.text.InsertSpeedTest.future:91 - future list.size: 50
2023-03-29 12:25:00,245 [INFO ] com.text.InsertSpeedTest.future:91 - future list.size: 50
2023-03-29 12:25:00,245 [INFO ] com.text.InsertSpeedTest.future:91 - future list.size: 50
2023-03-29 12:25:00,245 [INFO ] com.text.InsertSpeedTest.future:91 - future list.size: 50
2023-03-29 12:25:00,246 [INFO ] com.text.InsertSpeedTest.future:91 - future list.size: 50
2023-03-29 12:25:00,246 [INFO ] com.text.InsertSpeedTest.future:91 - future list.size: 50
2023-03-29 12:25:00,246 [INFO ] com.text.InsertSpeedTest.future:91 - future list.size: 50
2023-03-29 12:25:00,246 [INFO ] com.text.InsertSpeedTest.future:91 - future list.size: 50
2023-03-29 12:25:00,251 [INFO ] com.text.InsertSpeedTest.future:91 - future list.size: 50
2023-03-29 12:25:00,250 [INFO ] com.text.InsertSpeedTest.future:91 - future list.size: 50
2023-03-29 12:25:00,250 [INFO ] com.text.InsertSpeedTest.future:91 - future list.size: 50
2023-03-29 12:25:00,250 [INFO ] com.text.InsertSpeedTest.future:91 - future list.size: 50
2023-03-29 12:25:00,254 [INFO ] com.text.InsertSpeedTest.future:91 - future list.size: 50
2023-03-29 12:25:00,255 [INFO ] com.text.InsertSpeedTest.future:91 - future list.size: 50
2023-03-29 12:25:00,255 [INFO ] com.text.InsertSpeedTest.future:91 - future list.size: 50
2023-03-29 12:25:00,256 [INFO ] com.text.InsertSpeedTest.future:91 - future list.size: 50
2023-03-29 12:25:00,256 [INFO ] com.text.InsertSpeedTest.future:91 - future list.size: 50
2023-03-29 12:25:00,256 [INFO ] com.text.InsertSpeedTest.future:91 - future list.size: 50
2023-03-29 12:25:00,259 [INFO ] com.text.InsertSpeedTest.future:91 - future list.size: 50
2023-03-29 12:25:00,260 [INFO ] com.text.InsertSpeedTest.future:91 - future list.size: 50
2023-03-29 12:25:00,260 [INFO ] com.text.InsertSpeedTest.future:91 - future list.size: 50
2023-03-29 12:25:00,261 [INFO ] com.text.InsertSpeedTest.future:91 - future list.size: 50
2023-03-29 12:25:00,261 [INFO ] com.text.InsertSpeedTest.future:91 - future list.size: 50
2023-03-29 12:25:00,261 [INFO ] com.text.InsertSpeedTest.future:91 - future list.size: 50
2023-03-29 12:25:00,261 [INFO ] com.text.InsertSpeedTest.future:91 - future list.size: 50
2023-03-29 12:25:00,262 [INFO ] com.text.InsertSpeedTest.future:91 - future list.size: 50
2023-03-29 12:25:00,262 [INFO ] com.text.InsertSpeedTest.future:91 - future list.size: 50
2023-03-29 12:25:00,262 [INFO ] com.text.InsertSpeedTest.future:91 - future list.size: 50
2023-03-29 12:25:00,262 [INFO ] com.text.InsertSpeedTest.future:91 - future list.size: 50
2023-03-29 12:25:00,262 [INFO ] com.text.InsertSpeedTest.future:91 - future list.size: 50
2023-03-29 12:25:00,263 [INFO ] com.text.InsertSpeedTest.future:91 - future list.size: 50
2023-03-29 12:25:00,263 [INFO ] com.text.InsertSpeedTest.future:91 - future list.size: 50
2023-03-29 12:25:09,247 [INFO ] com.text.InsertSpeedTest.lambda$future$1:96 - Save Completed... list.size: 50, completedCount: 50, (1/60)
2023-03-29 12:25:09,248 [INFO ] com.text.InsertSpeedTest.lambda$future$1:96 - Save Completed... list.size: 50, completedCount: 100, (2/60)
2023-03-29 12:25:09,248 [INFO ] com.text.InsertSpeedTest.future:91 - future list.size: 50
2023-03-29 12:25:09,248 [INFO ] com.text.InsertSpeedTest.lambda$future$1:96 - Save Completed... list.size: 50, completedCount: 150, (3/60)
2023-03-29 12:25:09,248 [INFO ] com.text.InsertSpeedTest.lambda$future$1:96 - Save Completed... list.size: 50, completedCount: 200, (4/60)
2023-03-29 12:25:09,248 [INFO ] com.text.InsertSpeedTest.future:91 - future list.size: 50
2023-03-29 12:25:09,248 [INFO ] com.text.InsertSpeedTest.future:91 - future list.size: 50
2023-03-29 12:25:09,248 [INFO ] com.text.InsertSpeedTest.lambda$future$1:96 - Save Completed... list.size: 50, completedCount: 250, (5/60)
2023-03-29 12:25:09,248 [INFO ] com.text.InsertSpeedTest.future:91 - future list.size: 50
2023-03-29 12:25:09,248 [INFO ] com.text.InsertSpeedTest.lambda$future$1:96 - Save Completed... list.size: 50, completedCount: 300, (6/60)
2023-03-29 12:25:09,248 [INFO ] com.text.InsertSpeedTest.lambda$future$1:96 - Save Completed... list.size: 50, completedCount: 350, (7/60)
2023-03-29 12:25:09,249 [INFO ] com.text.InsertSpeedTest.lambda$future$1:96 - Save Completed... list.size: 50, completedCount: 400, (8/60)
2023-03-29 12:25:09,248 [INFO ] com.text.InsertSpeedTest.future:91 - future list.size: 50
2023-03-29 12:25:09,248 [INFO ] com.text.InsertSpeedTest.future:91 - future list.size: 50
2023-03-29 12:25:09,249 [INFO ] com.text.InsertSpeedTest.lambda$future$1:96 - Save Completed... list.size: 50, completedCount: 450, (9/60)
2023-03-29 12:25:09,249 [INFO ] com.text.InsertSpeedTest.lambda$future$1:96 - Save Completed... list.size: 50, completedCount: 500, (10/60)
2023-03-29 12:25:09,249 [INFO ] com.text.InsertSpeedTest.future:91 - future list.size: 50
2023-03-29 12:25:09,249 [INFO ] com.text.InsertSpeedTest.future:91 - future list.size: 50
2023-03-29 12:25:09,249 [INFO ] com.text.InsertSpeedTest.future:91 - future list.size: 50
2023-03-29 12:25:09,249 [INFO ] com.text.InsertSpeedTest.lambda$future$1:96 - Save Completed... list.size: 50, completedCount: 550, (11/60)
2023-03-29 12:25:09,249 [INFO ] com.text.InsertSpeedTest.lambda$future$1:96 - Save Completed... list.size: 50, completedCount: 600, (12/60)
2023-03-29 12:25:09,249 [INFO ] com.text.InsertSpeedTest.future:91 - future list.size: 50
2023-03-29 12:25:09,249 [INFO ] com.text.InsertSpeedTest.lambda$future$1:96 - Save Completed... list.size: 50, completedCount: 650, (13/60)
2023-03-29 12:25:09,249 [INFO ] com.text.InsertSpeedTest.future:91 - future list.size: 50
2023-03-29 12:25:09,249 [INFO ] com.text.InsertSpeedTest.lambda$future$1:96 - Save Completed... list.size: 50, completedCount: 700, (14/60)
2023-03-29 12:25:09,249 [INFO ] com.text.InsertSpeedTest.future:91 - future list.size: 50
2023-03-29 12:25:09,249 [INFO ] com.text.InsertSpeedTest.lambda$future$1:96 - Save Completed... list.size: 50, completedCount: 750, (15/60)
2023-03-29 12:25:09,249 [INFO ] com.text.InsertSpeedTest.future:91 - future list.size: 50
2023-03-29 12:25:09,249 [INFO ] com.text.InsertSpeedTest.lambda$future$1:96 - Save Completed... list.size: 50, completedCount: 800, (16/60)
2023-03-29 12:25:09,249 [INFO ] com.text.InsertSpeedTest.future:91 - future list.size: 50
2023-03-29 12:25:09,249 [INFO ] com.text.InsertSpeedTest.lambda$future$1:96 - Save Completed... list.size: 50, completedCount: 850, (17/60)
2023-03-29 12:25:09,250 [INFO ] com.text.InsertSpeedTest.future:91 - future list.size: 50
2023-03-29 12:25:09,250 [INFO ] com.text.InsertSpeedTest.future:91 - future list.size: 50
2023-03-29 12:25:09,250 [INFO ] com.text.InsertSpeedTest.lambda$future$1:96 - Save Completed... list.size: 50, completedCount: 900, (18/60)
2023-03-29 12:25:09,250 [INFO ] com.text.InsertSpeedTest.future:91 - future list.size: 50
2023-03-29 12:25:09,250 [INFO ] com.text.InsertSpeedTest.lambda$future$1:96 - Save Completed... list.size: 50, completedCount: 950, (19/60)
2023-03-29 12:25:09,250 [INFO ] com.text.InsertSpeedTest.lambda$future$1:96 - Save Completed... list.size: 50, completedCount: 1000, (20/60)
2023-03-29 12:25:09,250 [INFO ] com.text.InsertSpeedTest.future:91 - future list.size: 50
2023-03-29 12:25:09,250 [INFO ] com.text.InsertSpeedTest.lambda$future$1:96 - Save Completed... list.size: 50, completedCount: 1050, (21/60)
2023-03-29 12:25:09,250 [INFO ] com.text.InsertSpeedTest.lambda$future$1:96 - Save Completed... list.size: 50, completedCount: 1100, (22/60)
2023-03-29 12:25:09,250 [INFO ] com.text.InsertSpeedTest.future:91 - future list.size: 50
2023-03-29 12:25:09,250 [INFO ] com.text.InsertSpeedTest.lambda$future$1:96 - Save Completed... list.size: 50, completedCount: 1150, (23/60)
2023-03-29 12:25:09,250 [INFO ] com.text.InsertSpeedTest.future:91 - future list.size: 50
2023-03-29 12:25:09,250 [INFO ] com.text.InsertSpeedTest.lambda$future$1:96 - Save Completed... list.size: 50, completedCount: 1200, (24/60)
2023-03-29 12:25:09,250 [INFO ] com.text.InsertSpeedTest.future:91 - future list.size: 50
2023-03-29 12:25:09,250 [INFO ] com.text.InsertSpeedTest.future:91 - future list.size: 50
2023-03-29 12:25:09,250 [INFO ] com.text.InsertSpeedTest.lambda$future$1:96 - Save Completed... list.size: 50, completedCount: 1250, (25/60)
2023-03-29 12:25:09,251 [INFO ] com.text.InsertSpeedTest.future:91 - future list.size: 50
2023-03-29 12:25:09,251 [INFO ] com.text.InsertSpeedTest.lambda$future$1:96 - Save Completed... list.size: 50, completedCount: 1300, (26/60)
2023-03-29 12:25:09,251 [INFO ] com.text.InsertSpeedTest.lambda$future$1:96 - Save Completed... list.size: 50, completedCount: 1350, (27/60)
2023-03-29 12:25:09,251 [INFO ] com.text.InsertSpeedTest.future:91 - future list.size: 50
2023-03-29 12:25:09,251 [INFO ] com.text.InsertSpeedTest.lambda$future$1:96 - Save Completed... list.size: 50, completedCount: 1400, (28/60)
2023-03-29 12:25:09,252 [INFO ] com.text.InsertSpeedTest.lambda$future$1:96 - Save Completed... list.size: 50, completedCount: 1450, (29/60)
2023-03-29 12:25:09,252 [INFO ] com.text.InsertSpeedTest.future:91 - future list.size: 50
2023-03-29 12:25:09,252 [INFO ] com.text.InsertSpeedTest.future:91 - future list.size: 50
2023-03-29 12:25:15,620 [INFO ] com.text.InsertSpeedTest.lambda$future$1:96 - Save Completed... list.size: 50, completedCount: 1500, (30/60)
2023-03-29 12:25:15,620 [INFO ] com.text.InsertSpeedTest.lambda$future$1:96 - Save Completed... list.size: 50, completedCount: 1550, (31/60)
2023-03-29 12:25:15,620 [INFO ] com.text.InsertSpeedTest.lambda$future$1:96 - Save Completed... list.size: 50, completedCount: 1600, (32/60)
2023-03-29 12:25:15,620 [INFO ] com.text.InsertSpeedTest.lambda$future$1:96 - Save Completed... list.size: 50, completedCount: 1650, (33/60)
2023-03-29 12:25:15,620 [INFO ] com.text.InsertSpeedTest.lambda$future$1:96 - Save Completed... list.size: 50, completedCount: 1700, (34/60)
2023-03-29 12:25:15,620 [INFO ] com.text.InsertSpeedTest.future:91 - future list.size: 50
2023-03-29 12:25:15,620 [INFO ] com.text.InsertSpeedTest.lambda$future$1:96 - Save Completed... list.size: 50, completedCount: 1750, (35/60)
2023-03-29 12:25:15,620 [INFO ] com.text.InsertSpeedTest.lambda$future$1:96 - Save Completed... list.size: 50, completedCount: 1800, (36/60)
2023-03-29 12:25:15,620 [INFO ] com.text.InsertSpeedTest.lambda$future$1:96 - Save Completed... list.size: 50, completedCount: 1850, (37/60)
2023-03-29 12:25:15,620 [INFO ] com.text.InsertSpeedTest.lambda$future$1:96 - Save Completed... list.size: 50, completedCount: 1900, (38/60)
2023-03-29 12:25:15,620 [INFO ] com.text.InsertSpeedTest.future:91 - future list.size: 50
2023-03-29 12:25:15,620 [INFO ] com.text.InsertSpeedTest.lambda$future$1:96 - Save Completed... list.size: 50, completedCount: 1950, (39/60)
2023-03-29 12:25:15,620 [INFO ] com.text.InsertSpeedTest.lambda$future$1:96 - Save Completed... list.size: 50, completedCount: 2000, (40/60)
2023-03-29 12:25:15,620 [INFO ] com.text.InsertSpeedTest.lambda$future$1:96 - Save Completed... list.size: 50, completedCount: 2050, (41/60)
2023-03-29 12:25:15,620 [INFO ] com.text.InsertSpeedTest.lambda$future$1:96 - Save Completed... list.size: 50, completedCount: 2100, (42/60)
2023-03-29 12:25:15,620 [INFO ] com.text.InsertSpeedTest.lambda$future$1:96 - Save Completed... list.size: 50, completedCount: 2150, (43/60)
2023-03-29 12:25:15,620 [INFO ] com.text.InsertSpeedTest.lambda$future$1:96 - Save Completed... list.size: 50, completedCount: 2200, (44/60)
2023-03-29 12:25:15,621 [INFO ] com.text.InsertSpeedTest.lambda$future$1:96 - Save Completed... list.size: 50, completedCount: 2250, (45/60)
2023-03-29 12:25:15,621 [INFO ] com.text.InsertSpeedTest.lambda$future$1:96 - Save Completed... list.size: 50, completedCount: 2300, (46/60)
2023-03-29 12:25:15,621 [INFO ] com.text.InsertSpeedTest.lambda$future$1:96 - Save Completed... list.size: 50, completedCount: 2350, (47/60)
2023-03-29 12:25:15,621 [INFO ] com.text.InsertSpeedTest.lambda$future$1:96 - Save Completed... list.size: 50, completedCount: 2400, (48/60)
2023-03-29 12:25:15,621 [INFO ] com.text.InsertSpeedTest.lambda$future$1:96 - Save Completed... list.size: 50, completedCount: 2450, (49/60)
2023-03-29 12:25:15,621 [INFO ] com.text.InsertSpeedTest.lambda$future$1:96 - Save Completed... list.size: 50, completedCount: 2500, (50/60)
2023-03-29 12:25:15,621 [INFO ] com.text.InsertSpeedTest.lambda$future$1:96 - Save Completed... list.size: 50, completedCount: 2550, (51/60)
2023-03-29 12:25:15,621 [INFO ] com.text.InsertSpeedTest.lambda$future$1:96 - Save Completed... list.size: 50, completedCount: 2600, (52/60)
2023-03-29 12:25:15,622 [INFO ] com.text.InsertSpeedTest.lambda$future$1:96 - Save Completed... list.size: 50, completedCount: 2700, (54/60)
2023-03-29 12:25:15,622 [INFO ] com.text.InsertSpeedTest.lambda$future$1:96 - Save Completed... list.size: 50, completedCount: 2750, (55/60)
2023-03-29 12:25:15,621 [INFO ] com.text.InsertSpeedTest.lambda$future$1:96 - Save Completed... list.size: 50, completedCount: 2650, (53/60)
2023-03-29 12:25:15,622 [INFO ] com.text.InsertSpeedTest.lambda$future$1:96 - Save Completed... list.size: 50, completedCount: 2800, (56/60)
2023-03-29 12:25:16,299 [INFO ] com.text.InsertSpeedTest.lambda$future$1:96 - Save Completed... list.size: 50, completedCount: 2850, (57/60)
2023-03-29 12:25:16,402 [INFO ] com.text.InsertSpeedTest.lambda$future$1:96 - Save Completed... list.size: 50, completedCount: 2900, (58/60)
2023-03-29 12:25:16,543 [INFO ] com.text.InsertSpeedTest.lambda$future$1:96 - Save Completed... list.size: 50, completedCount: 2950, (59/60)
2023-03-29 12:25:16,547 [INFO ] com.text.InsertSpeedTest.lambda$future$1:96 - Save Completed... list.size: 50, completedCount: 3000, (60/60)
2023-03-29 12:25:16,547 [INFO ] com.text.InsertSpeedTest.test3:156 - Insert running time: 16s
parallel()
메소드를 추가해서 실행시켜 보면 병렬로 처리되어 속도는 A 방법map(ComputableFuture::join)
에서 Bloacking 되어 하나하나씩 수행됨Spring Batch로 DB Delete 일괄 삭제 테스트를 진행 하던 도중에 좀 느린거 같아서 진행 했던 테스트인데 좀 흥미로웠던거 같다.