![• References
‘[Spark] 데이터프레임’ http://12bme.tistory.com/307
‘IPython/Jupyter SQL Magic Functions for PySpark’ https://db-blog.web.cern.ch/blog/luca-canali/2016-11-ipythonjupyter-sql-magic-functions-pyspark
‘IPython magic functions for Pyspark Examples of shortcuts for executing SQL in Spark’
https://github.com/LucaCanali/Miscellaneous/blob/master/Pyspark_SQL_Magic_Jupyter/IPython_Pyspark_SQL_Magic.ipynb](https://image.slidesharecdn.com/pysparkdataframe0411-180410152626/85/PySpark-dataframe-21-320.jpg)
PySpark dataframe
- 2. 1. PySpark RDD Communication 2. Catalyst Optimizer 3. DataFrame을 이용한 PySpark Speed-up - 실습 - 4. 데이터프레임 생성하기 5. 데이터프레임 쿼리 6. RDD와 같이 작업 7. 데이터프레임 API로 쿼리 8. 스파크 SQL로 쿼리 9. 비행기록(On-time flight) 데이터프레임 사용하기
- 3. 1. PySpark RDD Communication RDD에서 쿼리를 실행하는 것은 자바 JVM 과 Py4J 사이의 Context switching과 Communications overhead를 필요로 함.
- 4. 1. PySpark RDD Communication
- 17. • A DataFrame is a distributed collection of data organized into named columns. It is conceptually equivalent to a table in a relational database or a data frame in R/Python, but with richer optimizations under the hood. DataFrames can be constructed from a wide array of sources such as: structured data files, tables in Hive, external databases, or existing RDDs. 3. DataFrame
- 18. • A Tale of Three Apache Spark APIs: RDDs, DataFrames, and Datasets When to use them and why 3. DataFrame
- 20. 이제부터는 Jupyter Notebook 에서 실습하기 WIKI LINK()에서 실습코드 Download 4. DataFrame 생성하기 5. DataFrame Query 6. RDD와 같이 작업 7. DataFrame API Query 8. Spark SQL Query 9. 비행기록(On-time flight) DataFrame 사용하기 https://github.com/drabastomek/learningPySpark/blob/master/Chapter03/LearningPySpark_Chapter03.ipynb https://github.com/donwany/Databricks/blob/master/notebooks/Users/theophilus.siameh.consultant%40nielsen.com/Master/Lesson-3.py
- 21. • References ‘[Spark] 데이터프레임’ http://12bme.tistory.com/307 ‘IPython/Jupyter SQL Magic Functions for PySpark’ https://db-blog.web.cern.ch/blog/luca-canali/2016-11-ipythonjupyter-sql-magic-functions-pyspark ‘IPython magic functions for Pyspark Examples of shortcuts for executing SQL in Spark’ https://github.com/LucaCanali/Miscellaneous/blob/master/Pyspark_SQL_Magic_Jupyter/IPython_Pyspark_SQL_Magic.ipynb