csapi 프로젝트 (8) - broker url 환경변수로 수정하도록 변경

https://not-to-be-reset.tistory.com/343 글에 작성한 대로, 모델 다운로드 링크가 만료되고 코드가 동작하지 않아서 다시 포스팅 합니다.
단일 인스턴스 부터 쿠버네티스 클러스터까지 확장해 나가려고 합니다.

개발환경

2023. 10. 8 기준 wsl 윈도우에서 진행

github 주소

https://github.com/kyeongjun-dev/csapi

커밋 : https://github.com/kyeongjun-dev/csapi/tree/c808ce9d005a5d5738419333c43750f7c2a042ed

---

Broker(rabbitmq) 정보 수정

기본으로 생성되는 계정이 ID guest, PASSWD guest인데, 이를 사용할 수는 없으니 환경변수로 이를 설정합니다.

celery/csapi/tasks.py 파일에서 app 객체 생성 부분을 변경합니다. (40 line)

import cv2
import numpy as np
from tensorflow.keras.models import load_model
import tensorflow as tf

class fashion_tools(object):
    def __init__(self,imageid,model,version=1.1):
        self.imageid = imageid
        self.model   = model
        self.version = version
        
    def get_dress(self,stack=False):
        """limited to top wear and full body dresses (wild and studio working)"""
        """takes input rgb----> return PNG"""
        name =  self.imageid
        file = cv2.imread(name)
        file = tf.image.resize_with_pad(file,target_height=512,target_width=512)
        rgb  = file.numpy()
        file = np.expand_dims(file,axis=0)/ 255.
        seq = self.model.predict(file)
        seq = seq[3][0,:,:,0]
        seq = np.expand_dims(seq,axis=-1)
        c1x = rgb*seq
        c2x = rgb*(1-seq)
        cfx = c1x+c2x
        dummy = np.ones((rgb.shape[0],rgb.shape[1],1))
        rgbx = np.concatenate((rgb,dummy*255),axis=-1)
        rgbs = np.concatenate((cfx,seq*255.),axis=-1)
        if stack:
            stacked = np.hstack((rgbx,rgbs))
            return stacked
        else:
            return rgbs

    def get_patch(self):
        return None

from celery import Celery, current_task

app = Celery('tasks')

import uuid

@app.task
def api(input_image_name, output_image_name):
    saved = load_model("./model.h5")
    api = fashion_tools(input_image_name, saved)
    image_ = api.get_dress(False)
    file_type = input_image_name.split('.')[1]
    cv2.imwrite(output_image_name, image_)
    #return current_task.request.id
    return input_image_name + output_image_name

 

django/config/settings.py 파일에서 환경변수로 broker 정보를 설정하도록 수정합니다. (14, 128~135)
os.getenv는 첫 번째 파라미터의 값이 존재하지 않으면, 두 번째 파라미터를 읽어옵니다.

"""
Django settings for config project.

Generated by 'django-admin startproject' using Django 3.2.20.

For more information on this file, see
https://docs.djangoproject.com/en/3.2/topics/settings/

For the full list of settings and their values, see
https://docs.djangoproject.com/en/3.2/ref/settings/
"""

from pathlib import Path
import os
# Build paths inside the project like this: BASE_DIR / 'subdir'.
BASE_DIR = Path(__file__).resolve().parent.parent


# Quick-start development settings - unsuitable for production
# See https://docs.djangoproject.com/en/3.2/howto/deployment/checklist/

# SECURITY WARNING: keep the secret key used in production secret!
SECRET_KEY = 'django-insecure-dpq95)7kn^32io0!_@a%-utk%pl%nxwit=2bxn+l-m4*loly#u'

# SECURITY WARNING: don't run with debug turned on in production!
DEBUG = True

ALLOWED_HOSTS = ["localhost"]


# Application definition

INSTALLED_APPS = [
    'django.contrib.admin',
    'django.contrib.auth',
    'django.contrib.contenttypes',
    'django.contrib.sessions',
    'django.contrib.messages',
    'django.contrib.staticfiles',
    'csapi',
]

MIDDLEWARE = [
    'django.middleware.security.SecurityMiddleware',
    'django.contrib.sessions.middleware.SessionMiddleware',
    'django.middleware.common.CommonMiddleware',
    'django.middleware.csrf.CsrfViewMiddleware',
    'django.contrib.auth.middleware.AuthenticationMiddleware',
    'django.contrib.messages.middleware.MessageMiddleware',
    'django.middleware.clickjacking.XFrameOptionsMiddleware',
]

ROOT_URLCONF = 'config.urls'

TEMPLATES = [
    {
        'BACKEND': 'django.template.backends.django.DjangoTemplates',
        'DIRS': [BASE_DIR / 'templates'],
        'APP_DIRS': True,
        'OPTIONS': {
            'context_processors': [
                'django.template.context_processors.debug',
                'django.template.context_processors.request',
                'django.contrib.auth.context_processors.auth',
                'django.contrib.messages.context_processors.messages',
            ],
        },
    },
]

WSGI_APPLICATION = 'config.wsgi.application'


# Database
# https://docs.djangoproject.com/en/3.2/ref/settings/#databases

DATABASES = {
    'default': {
        'ENGINE': 'django.db.backends.sqlite3',
        'NAME': BASE_DIR / 'db.sqlite3',
    }
}


# Password validation
# https://docs.djangoproject.com/en/3.2/ref/settings/#auth-password-validators

AUTH_PASSWORD_VALIDATORS = [
    {
        'NAME': 'django.contrib.auth.password_validation.UserAttributeSimilarityValidator',
    },
    {
        'NAME': 'django.contrib.auth.password_validation.MinimumLengthValidator',
    },
    {
        'NAME': 'django.contrib.auth.password_validation.CommonPasswordValidator',
    },
    {
        'NAME': 'django.contrib.auth.password_validation.NumericPasswordValidator',
    },
]


# Internationalization
# https://docs.djangoproject.com/en/3.2/topics/i18n/

LANGUAGE_CODE = 'en-us'

TIME_ZONE = 'UTC'

USE_I18N = True

USE_L10N = True

USE_TZ = True


# Static files (CSS, JavaScript, Images)
# https://docs.djangoproject.com/en/3.2/howto/static-files/

STATIC_URL = '/static/'

# Default primary key field type
# https://docs.djangoproject.com/en/3.2/ref/settings/#default-auto-field

DEFAULT_AUTO_FIELD = 'django.db.models.BigAutoField'

BROKER_ID = os.getenv('BROKER_ID', 'guest')
BROKER_PASSWORD = os.getenv('BROKER_PASSWORD', 'guest')
BROKER_URL = os.getenv('BROKER_URL', 'rabbit')
BROKER_PORT = os.getenv('BROKER_PORT', '5672')

CELERY_BROKER_URL = f'amqp://{BROKER_ID}:{BROKER_PASSWORD}@{BROKER_URL}:{BROKER_PORT}//'
CELERY_RESULT_BACKEND = 'rpc://'
CELERY_TIMEZONE = 'Asia/Seoul'

 

다음으로 docker-compose.yml 파일에서 각 컨테이너 마다 환경변수를 이용하여 정보를 설정하도록 environment: 블럭을 추가합니다. (기존 : guest, guest에서 user, password로 변경)

version: "3"
services:
  django:
    build:
      context: ./django
    command: python manage.py runserver 0.0.0.0:8080
    ports:
      - "8080:8080"
    volumes:
      - image_volume:/csapi/images
    environment:
      - BROKER_ID=user
      - BROKER_PASSWORD=password
      - BROKER_URL=rabbit
      - BROKER_PORT=5672

  worker:
    build:
      context: ./celery
    command: celery -A csapi.tasks worker --loglevel=info --max-tasks-per-child 1 -c 1
    volumes:
      - image_volume:/csapi/images
    environment:
      - CELERY_BROKER_URL=pyamqp://user:password@rabbit:5672//
      - CELERY_RESULT_BACKEND=rpc://

  rabbit:
    image: rabbitmq:3-management
    expose:
      - "5672"
    ports:
      - "15672:15672"
    environment:
      - RABBITMQ_DEFAULT_USER=user
      - RABBITMQ_DEFAULT_PASS=password

volumes:
  image_volume:

 

아래와 같이 docker compose 로그에서 celery가 ready 상태가 되면 정상동작 입니다.

csapi-worker-1  | [2023-10-07 15:16:42,743: INFO/MainProcess] mingle: all alone
csapi-worker-1  | [2023-10-07 15:16:42,788: INFO/MainProcess] celery@c7e666b29855 ready.

---

주의할점은 docker-compose.yml에서 스케일링 할때 ports로 설정되어 있으면 여러 컨테이너가 포트를 포워딩하게 됨으로 주의해야 합니다.
예를들어, rabbit의 15672 포트포워딩 부분을 expose로 수정합니다. 이는 docker-compose up --scale rabbit=3 로 스케일링 했을때, 스케일링을 실패하기 때문입니다.

  rabbit:
    image: rabbitmq:3-management
    expose:
      - "5672"
      - "15672:15672"
    environment:
      - RABBITMQ_DEFAULT_USER=user
      - RABBITMQ_DEFAULT_PASS=password