[ERROR] (Not Solved) Airflow HttpSensor 400 Client Error: Bad Request for url

Denver·2023년 1월 1일

400 API REST airflow apache error spotify

ERROR

목록 보기

5/6

0. 실행 환경

AWS EC2 t2.xlarge
OS : Red Hat 9.1
Python : 3.9
Airflow : 2.5.0

1. Code

with DAG(
    is_api_available = HttpSensor(
        task_id = 'is_api_available',
        http_conn_id = 'spotify_api',
        # method="GET",
        headers = {
            # 'Accept': 'application/json',
            # 'Content-Type': 'application/json',
            'Authorization': 
                'Bearer <MYTOKEN>',
        },
        
        request_params = {
            'q': 'BTS',
            'type': 'artist',
            'limit': '1',
        },
        method="GET",
        endpoint='v1/search'
    )

curl, request로 하면 결과가 나오는데 task 실행시키니까 400 에러가난다.
HttpSensor, SimpleHttpOperator 로 해봐도 안된다.
response_check=False를 넣어도 안된다.
providers/http/hooks/http.py, requests/models.py를봐도 response code를 확인하는거 밖에 없는데 원인을 모르겠다.
task정의에서 header를 제거해도 같은 오류가난다.

2. Connection

3. API Test

1) Curl 결과

2) Requests

import requests

headers = {
    'Accept': 'application/json',
    'Content-Type': 'application/json',
    'Authorization': 'Bearer <MYTOKEN>',
}

params = {
    'q': 'BTS',
    'type': 'artist',
    'limit': '1',
}

response = requests.get('https://api.spotify.com/v1/search', params=params, headers=headers)
print(response)

4. Error

requests.exceptions.HTTPError: 400 Client Error: Bad Request for url: https://api.spotify.com/v1/search?q=BTS&type=artist&limit=1

[2023-01-01 08:52:44,836] {http.py:122} INFO - Poking: v1/search
[2023-01-01 08:52:44,839] {base.py:73} INFO - Using connection ID 'spotify_api' for task execution.
[2023-01-01 08:52:44,840] {http.py:150} INFO - Sending 'GET' to url: https://api.spotify.com/v1/search
[2023-01-01 08:52:44,988] {http.py:163} ERROR - HTTP error: Bad Request
[2023-01-01 08:52:44,989] {http.py:164} ERROR - {
  "error": {
    "status": 400,
    "message": "Only valid bearer authentication supported"
  }
}
[2023-01-01 08:52:44,989] {taskinstance.py:1772} ERROR - Task failed with exception
Traceback (most recent call last):
  File "/home/ec2-user/.local/lib/python3.9/site-packages/airflow/providers/http/hooks/http.py", line 161, in check_response
    response.raise_for_status()
  File "/opt/anaconda/anaconda3/lib/python3.9/site-packages/requests/models.py", line 1021, in raise_for_status
    raise HTTPError(http_error_msg, response=self)
requests.exceptions.HTTPError: 400 Client Error: Bad Request for url: https://api.spotify.com/v1/search?q=BTS&type=artist&limit=1

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/ec2-user/.local/lib/python3.9/site-packages/airflow/sensors/base.py", line 199, in execute
    poke_return = self.poke(context)
  File "/home/ec2-user/.local/lib/python3.9/site-packages/airflow/providers/http/sensors/http.py", line 137, in poke
    raise exc
  File "/home/ec2-user/.local/lib/python3.9/site-packages/airflow/providers/http/sensors/http.py", line 124, in poke
    response = hook.run(
  File "/home/ec2-user/.local/lib/python3.9/site-packages/airflow/providers/http/hooks/http.py", line 151, in run
    return self.run_and_check(session, prepped_request, extra_options)
  File "/home/ec2-user/.local/lib/python3.9/site-packages/airflow/providers/http/hooks/http.py", line 204, in run_and_check
    self.check_response(response)
  File "/home/ec2-user/.local/lib/python3.9/site-packages/airflow/providers/http/hooks/http.py", line 165, in check_response
    raise AirflowException(str(response.status_code) + ":" + response.reason)
airflow.exceptions.AirflowException: 400:Bad Request
[2023-01-01 08:52:44,990] {taskinstance.py:1322} INFO - Marking task as FAILED. dag_id=nft-pipeline, task_id=is_api_available, execution_date=20230101T085244, start_date=, end_date=20230101T085244
Traceback (most recent call last):
  File "/home/ec2-user/.local/lib/python3.9/site-packages/airflow/providers/http/hooks/http.py", line 161, in check_response
    response.raise_for_status()
  File "/opt/anaconda/anaconda3/lib/python3.9/site-packages/requests/models.py", line 1021, in raise_for_status
    raise HTTPError(http_error_msg, response=self)
requests.exceptions.HTTPError: 400 Client Error: Bad Request for url: https://api.spotify.com/v1/search?q=BTS&type=artist&limit=1

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/ec2-user/.local/bin/airflow", line 8, in <module>
    sys.exit(main())
  File "/home/ec2-user/.local/lib/python3.9/site-packages/airflow/__main__.py", line 39, in main
    args.func(args)
  File "/home/ec2-user/.local/lib/python3.9/site-packages/airflow/cli/cli_parser.py", line 52, in command
    return func(*args, **kwargs)
  File "/home/ec2-user/.local/lib/python3.9/site-packages/airflow/utils/cli.py", line 108, in wrapper
    return f(*args, **kwargs)
  File "/home/ec2-user/.local/lib/python3.9/site-packages/airflow/cli/commands/task_command.py", line 576, in task_test
    ti.run(ignore_task_deps=True, ignore_ti_state=True, test_mode=True)
  File "/home/ec2-user/.local/lib/python3.9/site-packages/airflow/utils/session.py", line 75, in wrapper
    return func(*args, session=session, **kwargs)
  File "/home/ec2-user/.local/lib/python3.9/site-packages/airflow/models/taskinstance.py", line 1673, in run
    self._run_raw_task(
  File "/home/ec2-user/.local/lib/python3.9/site-packages/airflow/utils/session.py", line 72, in wrapper
    return func(*args, **kwargs)
  File "/home/ec2-user/.local/lib/python3.9/site-packages/airflow/models/taskinstance.py", line 1378, in _run_raw_task
    self._execute_task_with_callbacks(context, test_mode)
  File "/home/ec2-user/.local/lib/python3.9/site-packages/airflow/models/taskinstance.py", line 1524, in _execute_task_with_callbacks
    result = self._execute_task(context, task_orig)
  File "/home/ec2-user/.local/lib/python3.9/site-packages/airflow/models/taskinstance.py", line 1585, in _execute_task
    result = execute_callable(context=context)
  File "/home/ec2-user/.local/lib/python3.9/site-packages/airflow/sensors/base.py", line 199, in execute
    poke_return = self.poke(context)
  File "/home/ec2-user/.local/lib/python3.9/site-packages/airflow/providers/http/sensors/http.py", line 137, in poke
    raise exc
  File "/home/ec2-user/.local/lib/python3.9/site-packages/airflow/providers/http/sensors/http.py", line 124, in poke
    response = hook.run(
  File "/home/ec2-user/.local/lib/python3.9/site-packages/airflow/providers/http/hooks/http.py", line 151, in run
    return self.run_and_check(session, prepped_request, extra_options)
  File "/home/ec2-user/.local/lib/python3.9/site-packages/airflow/providers/http/hooks/http.py", line 204, in run_and_check
    self.check_response(response)
  File "/home/ec2-user/.local/lib/python3.9/site-packages/airflow/providers/http/hooks/http.py", line 165, in check_response
    raise AirflowException(str(response.status_code) + ":" + response.reason)
airflow.exceptions.AirflowException: 400:Bad Request