๐Ÿ’ป/MLOps

[Pinot] ์˜คํ”ˆ์†Œ์Šค ๊ธฐ์—ฌ ํ”„๋กœ์„ธ์Šค

ruhz 2023. 3. 5. 13:25

1. ๋“ค์–ด๊ฐ€๊ธฐ ์ „, 

1.1 ๊ฐœ๋ฐœ ํ™˜๊ฒฝ ์„ค์ •

 

[Pinot] Apache Pinot ๊ฐœ๋ฐœ ํ™˜๊ฒฝ

Apache Pinot ์šด์˜์ด ์•„๋‹Œ, ๊ฐœ๋ฐœ ๋ฐ ํ”„๋กœ์ ํŠธ ๊ธฐ์—ฌ๋ฅผ ๋ชฉ์ ์œผ๋กœ ๊ฐœ๋ฐœ ํ™˜๊ฒฝ์„ ์„ค์ •ํ•œ๋‹ค. WSL ํ™˜๊ฒฝ์„ ๊ธฐ์ค€์œผ๋กœ ํ•œ๋‹ค. Running Pinot locally - Apache Pinot Docs Note that some installations of the JDK do not contain the JNI binding

hhlab.tistory.com

1.2 ๋ชจ๋“ˆ ๋ฐ ์•„ํ‚คํ…์ณ 

pinot-common ํ”„๋กœ์ ํŠธ ์ „๋ฐ˜์˜ ๊ณตํ†ต ํด๋ž˜์Šค๋“ค์„ ์ œ๊ณต (config, metrics, metadata, pql.parsers, request, response, fs, ...)
pinot-trasnport Broker์˜ ๋ถ„์‚ฐ-์ˆ˜์ง‘์„ ์ฒ˜๋ฆฌํ•˜๋Š” ํด๋ž˜์Šค, Server์™€ Broker์˜ ์—ฐ๊ฒฐ์„ ์ฒ˜๋ฆฌํ•˜๋Š” nettty์˜ wrapper ํด๋ž˜์Šค ๋“ฑ
pinot-core ์„ธ๊ทธ๋จผํŠธ ์ฒ˜๋ฆฌ, ์ธ๋ฑ์Šค, ์ฟผ๋ฆฌ ๋“ฑ ๋ฐ์ดํ„ฐ์™€ ๊ด€๋ จ๋œ Pinot์˜ ํ•ต์‹ฌ ๊ธฐ๋Šฅ์„ ์ œ๊ณต

์œ„ ๋ชจ๋“ˆ๋“ค์ด ์ฃผ๋กœ ๊ธฐ๋ฐ˜์ด ๋˜๋ฉฐ, ๊ทธ ์œ„์— ๋‹ค์–‘ํ•œ ๋ชจ๋“ˆ๋“ค์ด ์Œ“์ด๊ฒŒ ๋œ๋‹ค.

  • pinot-server, pinot-controller, pinot-broker
  • pinot-minion, pinot-hadoop

 

2. Pinot ๊ธฐ์—ฌํ•˜๊ธฐ

2.1 ์ž‘์—… ํ๋ฆ„

2.1.1. Github ์ด์Šˆ ๋งŒ๋“ค๊ธฐ 

๋‹ค์Œ๊ณผ ๊ฐ™์€ ๋‚ด์šฉ์ด ํฌํ•จ๋˜์–ด์•ผ ํ•˜๋ฉฐ,  'PEP-Request' ๋ผ๋ฒจ(Pinot Enhancement Proposal)์„ ๋ถ™์—ฌ์•ผ ํ•œ๋‹ค. 

  • ๋ฌด์—‡์„ ํ•ด์•ผํ•˜๋Š”๊ฐ€?
  • ๊ธฐ๋Šฅ์ด ํ•„์š”ํ•œ ์ด์œ ๋Š” ๋ฌด์—‡์ธ๊ฐ€?
  • ์ดˆ๊ธฐ ์•„์ด๋””์–ด ๋ฐ ์ œ์•ˆ์ด ์žˆ๋Š”๊ฐ€?

PMC(Project Management Committee)๋Š” ์ด ์ œ์•ˆ์„ ๋ณด๊ณ  ์„ค๊ณ„๋ฌธ์„œ๊ฐ€ ํ•„์š”ํ• ์ง€, ๋‹จ์ˆœ PR๋กœ ํ•ด๊ฒฐ ๊ฐ€๋Šฅํ• ์ง€๋ฅผ ๊ฒฐ์ •ํ•˜๋ฉฐ, ๋ฆฌ๋ทฐ ๊นŒ์ง€ 5์ผ ์ •๋„์˜ ์‹œ๊ฐ„์ด ๊ฑธ๋ฆด ์ˆ˜ ์žˆ๋‹ค. PMC๋กœ ๋ถ€ํ„ฐ -1 ํ‘œ ์—†์ด, ํ•˜๋‚˜ ์ด์ƒ์˜ +1 ํ‘œ๋ฅผ ๋ฐ›๋Š”๋‹ค๋ฉด Github Issue์— ๋Œ€ํ•œ ๋งํฌ์™€ ํ•จ๊ป˜ PR์„ ์ œ์ถœํ•  ์ˆ˜ ์žˆ๋‹ค. 

 

GitHub - apache/pinot: Apache Pinot - A realtime distributed OLAP datastore

Apache Pinot - A realtime distributed OLAP datastore - GitHub - apache/pinot: Apache Pinot - A realtime distributed OLAP datastore

github.com

 

2.1.2. ์ด์Šˆ ๋ธŒ๋žœ์น˜ ๋งŒ๋“ค๊ธฐ

cd pinot
#
# ensure you are starting from the latest code base
# the following steps, ensure your fork's (origin's) master is up-to-date
#
git fetch upstream
git checkout master
git merge upstream/master
# create a branch for your issue
git checkout -b <your issue branch>

 

2.1.3. ์ฝ”๋“œ ์ˆ˜์ •ํ•˜๊ธฐ 

  • ์ฝ”๋“œ ๋ฌธ์„œํ™” : public ๋ฉ”์†Œ๋“œ์™€ ๋งค๊ฐœ๋ณ€์ˆ˜์— ๋Œ€ํ•ด, Class ๋ ˆ๋ฒจ์˜ Java docs๋ฅผ ํฌํ•จํ•ด์•ผ ํ•œ๋‹ค.
  • ๋กœ๊น… : slf4j ๋กœ๊ฑฐ๋ฅผ ์‚ฌ์šฉํ•œ๋‹ค.
  • ํ˜ธํ™˜์„ฑ : Zookeeper ๋˜๋Š” ์„ธ๊ทธ๋จผํŠธ์— ์ €์žฅ๋œ ์ƒํƒœ์™€ ๊ด€๋ จ๋œ ๋ณ€๊ฒฝ์€ ๋ฒ„์ „์˜ ํ˜ธํ™˜์„ฑ ๋ฌธ์ œ๊ฐ€ ์ค‘์š”ํ•˜๋‹ค.
    (ex. ๋ฒ„์ „์„ ์—…๋ฐ์ดํŠธ ํ•˜๋Š” ๊ฒฝ์šฐ, Server์™€ Broker๊ฐ€ ๋ฒ„์ „์ด ๋‹ค๋ฅธ ๊ฒฝ์šฐ ๋“ฑ)
  • ์™ธ๋ถ€ ๋ผ์ด๋ธŒ๋Ÿฌ๋ฆฌ
    • ํ˜„์žฌ ํ”„๋กœ์ ํŠธ์— ํฌํ•จ๋œ ๋ผ์ด๋ธŒ๋Ÿฌ๋ฆฌ ์ค‘ ๊ธฐ๋Šฅ์„ ๋Œ€์ฒดํ•  ์ˆ˜ ์žˆ๋Š” ๊ฒƒ์ด ์žˆ๋Š”๊ฐ€?
    • ํ•ด๋‹น ๋ผ์ด๋ธŒ๋Ÿฌ๋ฆฌ๋Š” ์ง€์† ์—…๋ฐ์ดํŠธ๊ฐ€ ๋˜๊ณ  ์žˆ๋Š”๊ฐ€?
    • ๋ผ์ด์„ ์Šค์˜ ๋ฌธ์ œ๋Š” ์—†๋Š”๊ฐ€?
  • ํ…Œ์ŠคํŠธ
    • ๋‹จ์œ„ ํ…Œ์ŠคํŠธ : positive, negative case ๋ชจ๋‘ ํฌํ•จํ•ด์•ผ ํ•˜๋ฉฐ, ํ…Œ์ŠคํŠธ์— ๋Œ€ํ•œ ์„ค๋ช…์ด ํฌํ•จ๋˜์–ด์•ผ ํ•œ๋‹ค.
    • IT ํ…Œ์ŠคํŠธ : Mockito๋กœ ๋ชจํ‚น์„ ์ˆ˜ํ–‰ํ•˜๋ฉฐ, REST API์˜ ๊ฐ ์—๋Ÿฌ ์ฝ”๋“œ์— ๋Œ€ํ•œ IT ํ…Œ์ŠคํŠธ๋Š” ํ•„์ˆ˜๋กœ ํฌํ•จ๋˜์–ด์•ผ ํ•œ๋‹ค.

 

2.1.4.  PR

# 0. ๋ผ์ด์„ ์Šค ํ—ค๋”๋ฅผ ๋ถ™์—ฌ์ค€๋‹ค.
mvn license:format

# 1. ์Šคํƒ€์ผ์„ ์ ๊ฒ€ํ•œ๋‹ค.
mvn checkstyle:check

# 2. ๋นŒ๋“œํ•˜๋ฉฐ, ํ…Œ์ŠคํŠธ๋ฅผ ๋Œ๋ ค๋ณธ๋‹ค. 
mvn clean install -Pbin-dist

# 3. Push ํ•œ๋‹ค. 
git add <files required for the change>
git commit -m "Meaningful oneliner for the change"
git push origin <your issue branch>

# After this, create a PullRequest in `github <https://github.com/apache/pinot/pulls>`_. Include the following information in the description:
#   * The changes that are included in the PR.
#   * Design document, if any.
#   * Information on any implementation choices that were made.
#   * Evidence of sufficient testing. You ``MUST`` indicate the tests done, either manually or automated.
#
# Once the PR is created, the code base is compiled and all tests are run via ``travis``. Make sure you followup on any issues flagged by travis and address them.
# If you see test failures that are intermittent, ``please`` create an issue to track them.
#
# Once the ``travis`` run is clear, request reviews from atleast 2 committers on the project and be sure to gently to followup on the issue with the reviewers.

# 4. PR ํ›„ ์ฝ”๋ฉ˜ํŠธ๋‚˜ ๋ณ€๊ฒฝ์‚ฌํ•ญ์ด ์กด์žฌํ•  ๋•Œ๋Š” ๋ฐ˜์˜ํ•ด์ค€๋‹ค. 
# commit your changes
git add <updated files>
git commit -m "Meaningful message for the udpate"
# pull new changes
git checkout master
git merge upstream/master
git checkout <your issue branch>
git rebase master

# At this time, if rebase flags any conflicts, resolve the conflicts and follow the instructions provided by the rebase command.
# Run additional tests/validations for the new changes and update the PR by pushing your changes:

git push origin <your issue branch>

 

2.1.5. ๋ฌธ์„œ ๋ฐ˜์˜

 

Update Documentation - Apache Pinot Docs

Pinot documentation is powered by Gitbook, and a bi-directional Github integration is set up to back up all the changes.

docs.pinot.apache.org

 

์Šฌ๋ž™ ์ฑ„๋„์— ์ฐธ์—ฌํ•  ์ˆ˜๋„ ์žˆ๋‹ค. 

 

Community Inviter - Auto Invitation for Slack

Automated way of increasing your slack community members. Your users will be able to sign up to your Slack community in a breeze.

communityinviter.com

'๐Ÿ’ป > MLOps' ์นดํ…Œ๊ณ ๋ฆฌ์˜ ๋‹ค๋ฅธ ๊ธ€

stern : ๋ถ„์‚ฐ ์ฒ˜๋ฆฌ ํ”Œ๋žซํผ ๋ชจ๋‹ˆํ„ฐ๋ง  (0) 2024.02.23
[k8s] `kubectl top`  (0) 2024.01.02
Apache Cassandra  (0) 2023.07.31
[Pinot] Apache Pinot ๊ฐœ๋ฐœ ํ™˜๊ฒฝ  (0) 2023.03.05
[Pinot] Apache Pinot?  (0) 2022.11.05