跳到主要內容

簡易檢索 / 詳目顯示

回結果列表
研究生： 呂銘洋
Ming-Yang Lu
論文名稱： 基於替代語意的 pandas DataFrame 靜態型別檢查器
An Alternative Semantic-based Type Checker for pandas Dataframes
指導教授： 莊永裕
口試委員:
學位類別： 碩士
Master
系所名稱： 資訊電機學院 - 資訊工程學系
Department of Computer Science & Information Engineering
論文出版年： 2020
畢業學年度： 108
語文別： 中文
論文頁數： 80
中文關鍵詞： 程式語言 、資料框 、資料科學 、靜態分析
相關次數： 點閱：7  下載：0 
分享至:
查詢本校圖書館目錄 查詢臺灣博碩士論文知識加值系統 勘誤回報

    • DataFrame 在資料科學中經常被用來處理表狀資料的概念,Python
    的 pandas 函數庫是一個廣被實用的 DataFrame 實作。而因為受限於複
    雜的 API 設計與缺乏靜態工具,使用者在編寫 pandas 程式時往往是容易
    犯錯的,這些錯誤來自於:沒有追蹤行欄位的標籤與型別、透過字串傳
    遞的旗標參數與缺乏函數參數的型別資訊。
    本研究討論了為 pandas 提供靜態分析功能的難處,並提出了一個用
    於靜態分析 pandas 程式的靜態型別檢查器:我們提出的作法基於使用替
    代語意,並且我們討論了如何為常用的 API 進行靜態檢查。並且我們使
    用 Python 完成了一個初步實作以作為概念驗證。我們透過定量分析的
    方式評估我們初步實作的功能性,並以使用案例來討論我們的工具能如
    何被使用。

    Dataframe is a well-used concept in Data Science tasks, which
    makes abstract on how a programmer manipulates tabular data. pandas
    is a popular and widely-used Python library which implements dataframes.
    Due to the complexity of its API design and lacking static analysis tools,
    programming in pandas is considerably an error-prone task. There are
    three types of common error: errors due to lacking column labels and
    types, errors due to string-typed flag arguments, and errors due to
    lacking type informations of functions as arguments.
    In this paper, we discuss what is the challenges of providing the
    ability of static analysis on pandas programs. We propose a static type
    checker for pandas programs based on Alternative Semantic. We also dis-
    cuss how to statically analyze common pandas APIs.
    We develop a preliminary implementation based on Python as our
    proof of concept. We evaluate our implementation with a qualitative
    analysis on its functionality and we discuss a case study about how can
    our checker reduce error‐proneness during development.

    摘要 (p.xiii) Abstract (p.xv) 目錄 (p.xvii) 圖目錄 (p.xxi) 表目錄 (p.xxiii) 1 緒論 (p.1) 1.1 DataFrame (p.1) Pandas 中的 DataFrame (p.1) 1.2 Type Checking (p.3) 2 研究動機 (p.5) 2.1 範例 (p.5) 2.2 幫 pandas 做這件事的好處 (p.8) 減少學習成本 (p.8) 減少重用程式碼時所遇到的問題 (p.8) Python 具有較高的使用彈性 (p.8) 有能力提供更好的開發環境 (p.8) 2.3 幫 pandas 做這件事的難點 (p.8) DataFrame 本身缺乏研究 (p.8) 現有外部檢查器的型別系統不夠強 (p.9) 3 提案 (p.11) 3.1 靜態檢查 (p.11) 3.2 模型建立 (p.13) 3.2.1 為 DataFrame 建立模型 (p.13) 3.2.2 為 Series 建立模型 (p.14) 3.2.3 元素計算 (p.14) 3.2.4 處理賦值動作 (p.15) 3.2.5 為標籤建立模型 (p.16) 3.2.6 為 melt / pivot 建立模型 (p.16) 3.2.7 為 apply / agg 建立模型 (p.17) 3.2.8 為 GroupBy / window 建立模型 (p.18) 3.3 作法限制 (p.19) 3.3.1 推理資料型別 (p.19) 3.3.2 行欄位取決於 DataFrame 數值 (p.20) 3.3.3 行欄位取決於 Python 的值 (p.21) 3.3.4 通用的型別檢查 (p.21) 3.4 回到研究動機 (p.21) 4 實作 (p.23) 4.1 實作概觀 (p.23) 4.2 直譯器與檢查語意 (p.25) 4.3 檢查函數 (p.27) 4.3.1 字面值與常數 (p.27) 4.3.2 pd.read\protect \TU\textunderscore csv (p.28) 4.3.3 DataFrame 與 Series (p.28) 4.3.4 loc, iloc 等屬性 (p.29) 4.3.5 錯誤處理 (p.29) 4.4 檢查器前端 (p.29) 4.4.1 作為語言伺服器使用 (p.30) 4.5 檢查器語意 (p.31) 5 評估與討論 (p.33) 5.1 功能性 (p.33) 5.1.1 評估標準 (p.33) 檢查行標籤與型別 (FI) (p.33) 檢查旗標參數(flag arguments)是否正確 (FL) (p.34) 檢查輸入函數的輸入輸出型別 (FN) (p.34) 5.1.2 本研究實作的功能性 (p.34) 資料載入 (p.35) Series 與運算操作 (p.35) 賦值操作 (p.35) Multi-Level Index (p.35) stack/unstack (p.35) 與 DataFrame 本身無關的功能性 (p.35) 5.1.3 與相關實作的功能性比較 (p.37) Gamma (p.38) 功能性的方面 (p.38) Frames (p.38) 功能性的方面 (p.38) frameless (p.38) 功能性的方面 (p.39) 5.2 案例分析 (p.39) 協助重用既有的程式 (p.39) 6 相關研究 (p.43) Type Systems (p.43) 依存型別(Dependant Type) (p.43) Row polymorphism (p.43) Type-Level Programming (p.43) DSL (p.44) 特定領域語言(Domain-Specific Languages, DSL) (p.44) 7 總結 (p.45) 未來展望 (p.46) 參考文獻 (p.47)

    Andrzejak, A. et al. 2019. Agile construction of data science DSLs (tool demo). Proceedings of the 18th ACM SIGPLAN International Conference on Generative Programming: Concepts and Experiences (Athens, Greece, Oct. 2019), 27–33.
    Armbrust, M. et al. 2015. Spark SQL: Relational Data Processing in Spark. Proceedings of the 2015 ACM SIGMOD International Conference on Management of Data (Melbourne, Victoria, Australia, May 2015), 1383–1394.
    Chambers, J.M. and Hastie, T.J. 1991. Statistical Models in S. CRC Press, Inc.
    Codd, E.F. 1970. A relational model of data for large shared data banks. Communications of the ACM. 13, 6 (Jun. 1970), 377–387.
    van Deursen, A. et al. 2000. Domain-specific Languages: An Annotated Bibliography. SIGPLAN Not. 35, 6 (Jun. 2000), 26–36.
    Frames.CSV: 2020. https://hackage.haskell.org/package/Frames-0.6.4/docs/Frames-CSV.html. Accessed: 2020-05-10.
    How pandas infers data types when parsing CSV files: 2018. https://rushter.com/blog/pandas-data-type-inference/. Accessed: 2020-05-31.
    Hudak, P. 1996. Building Domain-specific Embedded Languages. ACM Comput. Surv. 28, 4es (Dec. 1996).
    jsiek 2014. What is Gradual Typing | Jeremy Siek.
    json-autotype: Automatic type declaration for JSON input data: //hackage.haskell.org/package/json-autotype. Accessed: 2020-05-10.
    Kiselyov, O. et al. 2010. Fun with type functions. Reflections on the work of CAR hoare. Springer. 301–331.
    Kluyver, T. et al. 2016. Jupyter Notebooks – a publishing format for reproducible computational workflows. Positioning and power in academic publishing: Players, agents and agendas (2016), 87–90.
    Ma, E.J. et al. 2019. pyjanitor: A cleaner aPI for cleaning data. Proceedings of the 18th Python in Science Conference (2019), 50–53.
    Martin-Löf, P. 1975. An Intuitionistic Theory of Types: Predicative Part. Studies in Logic and the Foundations of Mathematics. H.E. Rose and J.C. Shepherdson, eds. Elsevier. 73–118.
    microsoft/pyright: 2020. https://github.com/microsoft/pyright. Accessed: 2020-06-22.
    Milner, R. 1978. A theory of type polymorphism in programming. Journal of Computer and System Sciences. 17, (1978), 348–375.
    mypy - Optional Static Typing for Python: http://mypy-lang.org/. Accessed: 2020-05-30.
    openlawlibrary/pygls: 2020. https://github.com/openlawlibrary/pygls. Accessed: 2020-05-24.
    PEP 484 – Type Hints: https://www.python.org/dev/peps/pep-0484/. Accessed: 2020-05-13.
    Petersohn, D. et al. 2020. Towards Scalable Dataframe Systems. arXiv:2001.00888 [cs]. (Jan. 2020).
    Petricek, T. 2017. Data exploration through dot-driven development. European Conference on Object-Oriented Programming (Barcelona, Spain, 2017).
    Petricek, T. et al. 2016. Types from data: making structured data first-class citizens in F#. Proceedings of the 37th ACM SIGPLAN Conference on Programming Language Design and Implementation (Santa Barbara, CA, USA, Jun. 2016), 477–490.
    Proof of Concept: TypedDataFrame · GitBook: https://typelevel.org/frameless/TypedDataFrame.html. Accessed: 2020-06-22.
    Pyre · A performant type-checker for Python 3: https://pyre-check.org/. Accessed: 2020-06-22.
    quicktype/quicktype: https://github.com/quicktype/quicktype. Accessed: 2020-05-10.
    R Core Team 2020. R: A language and environment for statistical computing.
    Siek, J.G. and Taha, W. 2006. Gradual Typing for Functional Languages. (2006).
    Syme, D. et al. 2013. Themes in information-rich functional programming for internet-scale data sources. Proceedings of the 2013 workshop on Data driven functional programming (Rome, Italy, Jan. 2013), 1–4.
    team, T. pandas development 2020. pandas-dev/pandas: Pandas. (Feb. 2020).
    typelevel/frameless: https://github.com/typelevel/frameless. Accessed: 2020-05-12.
    Wand, M. 1989. Type inference for record concatenation and multiple inheritance. [1989] Proceedings. Fourth Annual Symposium on Logic in Computer Science (Jun. 1989), 92–97.
    Wes McKinney 2010. Data Structures for Statistical Computing in Python. Proceedings of the 9th Python in Science Conference (2010), 56–61.
    Wu, Y. 2016. Is a Dataframe Just a Table? PLATEAU (2016), 10.

    QR CODE
    :::