跳到主要內容

簡易檢索 / 詳目顯示

回結果列表
研究生： 張桓融
Huan-Jung Chang
論文名稱： 強化 Venom Programming 的即時視覺處理 SDK
指導教授： 鄭永斌
Yung-Pin Cheng
口試委員:
學位類別： 碩士
Master
系所名稱： 資訊電機學院 - 資訊工程學系
Department of Computer Science & Information Engineering
論文出版年： 2025
畢業學年度： 113
語文別： 中文
論文頁數： 80
中文關鍵詞： Venom Programming 、工作流 、軟體擴充
外文關鍵詞： Venom Programming, workflow, software extension
相關次數： 點閱：82  下載：0 
分享至:
查詢本校圖書館目錄 查詢臺灣博碩士論文知識加值系統 勘誤回報

    • 軟體擴充通常依賴於原始程式碼的存取權限或是預先設計的擴充框架。然而，在無任何官方擴充手段的情況下，本實驗室開發的 Venom Programming（舊稱 Parasite Programming）技術已經能夠透過 SDK 取得應用程式的 UI 資訊，並利用透明視窗達成軟體擴充的效果。然而，現代應用程式往往包含影片播放、圖片、以及其他動態的渲染機制，導致無法單純使用抓取UI的方式進行分析。為了充分發揮 Venom Programming 在這些複雜應用場景中的潛力，強化其動態內容處理能力變得至關重要。
    本論文透過整合電腦視覺技術，大幅強化了 Venom Programming 的處理能力。在原有架構基礎上新增的跨語言電腦視覺後端，結合了 OpenCV 影像處理技術與 ONNX 模型推論能力，並採用工作流機制讓開發者能夠靈活組合各種視覺分析功能。透過這個創新架構，開發者能夠在不依賴額外工具的情況下，從目標應用程式中取得動態內容的資訊，並實時追蹤其變化狀態。這項技術突破顯著擴大了 Venom Programming 的應用範圍，使其能夠處理更加複雜的動態內容擴充需求。

    Software extension usually relies on access to source code or pre-designed extension frameworks. However, our laboratory has developed Venom Programming (formerly Parasite Programming) technology without any official extension methods. This technology can obtain application UI information through SDK and achieve software extension using transparent window technology.
    However, modern applications often contain video playback, images, and other dynamic rendering mechanisms. UI extraction methods alone cannot perform analysis on these contents. To make full use of Venom Programming in these complex application scenarios, we need to strengthen its dynamic content processing capabilities.
    This thesis enhances Venom Programming's processing capabilities by integrating computer vision technologies. We added a cross-language computer vision backend to the original architecture. This backend combines OpenCV image processing technology with ONNX model inference capabilities. It uses a workflow mechanism. Developers can combine various visual analysis functions through this mechanism. Developers can obtain dynamic content information from target applications through this architecture. They can track changes in real-time without additional tools. This enhancement expands the application scope of Venom Programming. It enables the technology to handle more complex dynamic content extension requirements.

    摘要 i Abstract ii 目錄 iii 圖目錄 vi 表目錄 viii 一、 緒論 1 二、 研究背景與相關技術 6 2-1 軟體擴充技術概述 6 2-1-1 傳統軟體擴充方法 6 2-1-2 RPA 7 2-1-3 UI Automation 8 2-1-4 Venom Programming 8 2-2 電腦視覺相關技術 10 2-2-1 OpenCV 影像處理 10 2-2-2 深度學習模型整合與 ONNX 格式支援 11 2-3 即時通訊技術 13 2-3-1 Polling 13 2-3-2 Long Polling 14 2-3-3 WebSocket 14 2-3-4 小結 15 2-4 Grafana Alloy 16 三、 支援Venom Programming的電腦視覺處理能力 18 3-1 Venom Programming應用場景分析 18 3-2 Venom Programming的影像擷取 21 3-2-1 螢幕擷取 21 3-2-2 視窗擷取 23 3-3 CV-Core電腦視覺後端 24 3-3-1 Google Meet手語識別需求分析 25 3-3-2 戰地風雲小地圖功能需求分析 26 3-3-3 戰地風雲物件辨識功能需求分析 28 3-3-4 使用構想 29 四、 電腦視覺後端架構 37 4-1 Venom Programming架構整合 37 4-1-1 原有Venom Programming架構回顧 37 4-1-2 CV-Core整合方式 39 4-2 CV-Core使用 40 4-2-1 工作流配置 41 4-2-2 工作流呼叫方式 43 4-2-3 與Venom Programming搭配 45 4-3 CV-Core整體設計 46 4-3-1 系統架構概述 46 4-3-2 Workflow Manager工作流管理實作 47 4-3-3 Workflow工作流實作 48 4-3-4 BaseTask、BaseTaskInput、BaseTaskOutput 50 4-4 工作流的限制 51 4-4-1 已實作視覺任務 52 4-4-2 OpenCV直接存取 54 五、 系統實現與評估 57 5-1 Battlefield Venom 57 5-2 功能性評估 58 5-3 效能評估 60 5-3-1 Websocket連線數量的效能分析 61 5-3-2 工作流分析 63 六、 結論與未來展望 66 七、 參考資料 67

    R. Kazman, S. Echeverría, and J. Ivers, "Extensibility," Carnegie Mellon University, Software Engineering Institute's Digital Library, 2022/04// 2022. [Online]. Available: https://doi.org/10.1184/R1/18863639
    [2] M. Lamothe, Y.-G. Guéhéneuc, and W. Shang, "A Systematic Review of API Evolution Literature," ACM Comput. Surv., vol. 54, no. 8, p. Article 171, 2021, doi: 10.1145/3470133.
    [3] A. Bisacchi, C. Caini, and S. Lanzoni, "Design and Implementation of a Bundle Protocol Unified API," in 2022 11th Advanced Satellite Multimedia Systems Conference and the 17th Signal Processing for Space Communications Workshop (ASMS/SPSC), 6–8 Sept. 2022 2022, pp. 1–6, doi: 10.1109/ASMS/SPSC55670.2022.9914734.
    [4] M. Aly, A. Charfi, and M. Mezini, "Building extensions for applications: Towards the understanding of extension possibilities," in 2013 21st International Conference on Program Comprehension (ICPC), 20–21 May 2013 2013, pp. 182–191, doi: 10.1109/ICPC.2013.6613846.
    [5] M. A. Sayed and N. Anjum, "Plug-In Architecture as Software Extension Mechanism: An Extensive Study of Eclipse Architecture."
    [6] W. M. P. van der Aalst, M. Bichler, and A. Heinzl, "Robotic Process Automation," Business & Information Systems Engineering, vol. 60, no. 4, pp. 269–272, 2018/08/01 2018, doi: 10.1007/s12599-018-0542-4.
    [7] "UiPath [Online]." https://www.uipath.com (accessed.
    [8] R. Haverty, "New accessibility model for Microsoft Windows and cross platform development," SIGACCESS Access. Comput., no. 82, pp. 11–17, 2005, doi: 10.1145/1077238.1077240.
    [9] T. Bi, X. Xia, D. Lo, J. Grundy, T. Zimmermann, and D. Ford, "Accessibility in Software Practice: A Practitioner’s Perspective," ACM Trans. Softw. Eng. Methodol., vol. 31, no. 4, p. Article 66, 2022, doi: 10.1145/3503508.
    [10] 陳宏輔, "利用軟體UI實現擴充功能突破原始碼限制的工具," Master, 資訊工程學系, 國立中央大學, 2023.
    [11] 劉蓉綺, "Windows桌面應用程式的擴增實境," Master, 資訊工程學系, 國立中央大學, 2022.
    [12] 鄧翔允, "無原始碼的軟體擴充技術：Parasite Programming," Master, 資訊工程學系, 國立中央大學, 2024.
    [13] 林欣蓓, "實現Parasite SDK中跨語言支持的後端解決方案," Master, 資訊工程學系, 國立中央大學, 2024.
    [14] C. Yung-Pin, L. Ching-Wei, and C. Yi-Cheng, "Apply computer vision in GUI automation for industrial applications," Mathematical Biosciences and Engineering, vol. 16, no. 6, pp. 7526–7545, 2019, doi: 10.3934/mbe.2019378.
    [15] C. Paduraru, M. Paduraru, and A. Stefanescu, "Automated game testing using computer vision methods," in 2021 36th IEEE/ACM International Conference on Automated Software Engineering Workshops (ASEW), 15–19 Nov. 2021 2021, pp. 65–72, doi: 10.1109/ASEW52652.2021.00024.
    [16] "OpenCV: Introduction [Online]." https://docs.opencv.org/4.11.0/d1/dfb/intro.html (accessed.
    [17] "ONNX [Online]." https://onnx.ai (accessed.
    [18] "Known Issues and Best Practices for the Use of Long Polling and Streaming in Bidirectional HTTP [Online]." https://www.rfc-editor.org/rfc/rfc6202 (accessed.
    [19] "The WebSocket Protocol [Online]." https://www.rfc-editor.org/rfc/rfc6455.html (accessed.
    [20] "Grafana Alloy [Online]." https://grafana.com/docs/alloy/latest/ (accessed.

    QR CODE
    :::