API 호출, 크롤링 우회

박우영·2023년 8월 6일

트러블 슈팅

목록 보기

17/19

개요

프로젝트는 특정 api, 정보에 대해 의존적입니다. 따라서 해당 정보를 가져와야하는데 제 개인 Local 환경에서는 잘 되지만 GKE 환경에서 되지 않는 Issue 가 발생하였습니다.

Error Msg

[http-nio-8084-exec-1] [2023-08-05 23:40:28,741] [ERROR] [SolvedCrawling.java:136] - <html><head><meta http-equiv="Content-Type" content="text/html; charset=iso-8859-1">
<title>ERROR: The request could not be satisfied</title>
</head><body>
<h1>403 ERROR</h1>
<h2>The request could not be satisfied.</h2>
<hr noshade="" size="1px">
Request blocked.
We can't connect to the server for this app or website at this time. There might be too much traffic or a configuration error. Try again later, or contact the app or website owner.
<br clear="all">
If you provide content to customers through CloudFront, you can find steps to troubleshoot and help prevent this error by reviewing the CloudFront documentation.
<br clear="all">
<hr noshade="" size="1px">
<pre>Generated by cloudfront (CloudFront)
Request ID: -ok_4ED_rixCpeCLsp6ytvEtEjyMwZSMUb2VxVa10USNMDizAGzXbg==
</pre>
<address>
</address>
</body></html>

Code

RestTemplate 설정

@Bean
public RestTemplate restTemplate() {
    RestTemplate restTemplate;
    try {
        restTemplate = new RestTemplate(clientHttpRequestFactory());
    }catch (Exception e){
        restTemplate = new RestTemplate();
    }
    restTemplate.setInterceptors(Collections.singletonList(
            (request, body, execution) -> {
                HttpHeaders headers = request.getHeaders();
                headers.setContentType(APPLICATION_JSON);
                headers.setAccept(Collections.singletonList(APPLICATION_JSON));
                headers.add(HttpHeaders.ACCEPT, "application/json");
                headers.add(HttpHeaders.USER_AGENT, "Mozilla/5.0");
                return execution.execute(request, body);
            }
    ));
    return restTemplate;
}

private HttpComponentsClientHttpRequestFactory clientHttpRequestFactory() {
    return new HttpComponentsClientHttpRequestFactory();
}

API 요청

public String getSubject(int problemId) throws Exception{
    String jsonString = null;

    try {
        jsonString = restTemplate.getForObject("https://solved.ac/api/v3/problem/show?problemId=" + problemId, String.class);
    }catch (Exception e){
        e.printStackTrace();
        throw new HttpResponseException("문제 정보를 가져오는데 실패했습니다.");
    }

    JSONParser jsonParser = new JSONParser();
    Object jsonObject = null;
    try {
        jsonObject = jsonParser.parse(jsonString);
    } catch (ParseException e) {
        e.printStackTrace();
    }

    JSONObject jsonBody = (JSONObject) jsonObject;

    return jsonBody.get("titleKo").toString();
}

크롤링

public BaekJoonDto profileCrawling(String baekjoonId) throws IOException, InterruptedException {
    WebDriver driver = setDriver();
    sleep(1000);
    driver.get(SOLVED_BASE_URL + SOLVED_PROFILE + baekjoonId);

    By solvedListBy = By.xpath("//*[@id=\"__next\"]/div[3]/div/div[6]/div[3]/div/table/tbody");
    sleep(1000);
    try {
        wait(driver, solvedListBy);
    } catch (TimeoutException | NoSuchElementException e) {
        log.error("{}", driver.getCurrentUrl());
        log.error("{}", driver.getPageSource());
        throw new CrawlingException("사용자 정보를 찾을 수 없습니다. 다시 시도해 주세요");
    }
    WebElement elements = driver.findElement(solvedListBy);
    WebElement webElement = elements.findElement(By.className("css-1ojb0xa"));
    int bronze = getUserSolvedCount(webElement, By.xpath("//*[@id=\"__next\"]/div[3]/div/div[6]/div[3]/div/table/tbody/tr[1]/td[2]/b"));

    driver.quit();
    return new BaekJoonDto(bronze);