PyTorch (Japanese Wikipedia)

PETALS: Collaborative Inference and Fine-tuning of Large Models p.2 Alexander Borzunov et al. 2023年
FlexGen: High-Throughput Generative Inference of Large Language Models with a Single GPU p.6 Ying Sheng et al. 2023年
"torch.fx, a program capture and transformation library for PyTorch" Reed, et al. (2021). Torch.fx: Practical Program Capture and Transformation for Deep Learning in Python. arxiv.
"torch.fx represents programs in a DAG-based IR" Reed, et al. (2021). Torch.fx: Practical Program Capture and Transformation for Deep Learning in Python. arxiv.
"graph-based ... IR ... Program transformations ... is as simple embedded programming languages that are meta-programmed from a host language, predominantly Python" Reed, et al. (2021). Torch.fx: Practical Program Capture and Transformation for Deep Learning in Python. arxiv.
"torch.fx provides an fx.graph_drawer package, which gives the user the ability to visualize torch.fx graphs with Graphviz" Reed, et al. (2021). Torch.fx: Practical Program Capture and Transformation for Deep Learning in Python. arxiv.
"Quantization makes use of torch.fx’s graph and GraphModule representation to simultaneously modify the program code and weight values." Reed, et al. (2021). Torch.fx: Practical Program Capture and Transformation for Deep Learning in Python. arxiv.
"6.2.2 Fusion Optimizations ... torch.fx provides the necessary non-local program context and state modification facilities needed for this transformation with its ahead-of-time, graph-based nature" Reed, et al. (2021). Torch.fx: Practical Program Capture and Transformation for Deep Learning in Python. arxiv.
" The project was quickly developed using torch.fx’s Python APIs as well as TensorRT’s Python APIs, creating a translation layer between the two." Reed, et al. (2021). Torch.fx: Practical Program Capture and Transformation for Deep Learning in Python. arxiv.
"the purposes of serialization or export. For instance, TorchScript" Reed, et al. (2021). Torch.fx: Practical Program Capture and Transformation for Deep Learning in Python. arxiv.

chainer.org

Chainer Team (2010年12月5日). “Chainer/CuPy v7のリリースと今後の開発体制について”. Chainer Blog. 2020年8月10日閲覧。

cloudflare.com

blog.cloudflare.com

Cloudflare R2 and MosaicML enable training LLMs on any compute, anywhere in the world, with zero switching costs Cloudflare 2023年5月16日

deepspeed.ai

DeepSpeed-MII: instant speedup on 24,000+ open-source DL models with up to 40x cheaper inference Microsoft 2022年10月10日
ZeRO++ Microsoft
ZeRO-Offload Microsoft
Pipeline Parallelism - DeepSpeed Microsoft
Mixture of Experts - DeepSpeed Microsoft
ZeRO-Inference: Democratizing massive model inference Microsoft 2022年9月9日

deepspeed.readthedocs.io

ZeRO Microsoft

dl4nlp.info

“Natural Language Processing (NLP) with PyTorch — NLP with PyTorch documentation” (英語). dl4nlp.info. 2019年3月30日閲覧。

doi.org

Ketkar, Nikhil (2017). “Introduction to PyTorch” (英語). Deep Learning with Python. Apress, Berkeley, CA. pp. 195–208. doi:10.1007/978-1-4842-2766-4_12. ISBN 9781484227657

fairscale.readthedocs.io

Efficient Memory management - FairScale documentation Facebook Research
Pipeline Parallelism - FairScale documentation Facebook Research

github.com

"PyTorch 2.6.0 Release"; 閲覧日: 2025年2月2日; 出版日: 2025年1月29日.
GitHub - catalyst-team/catalyst: Accelerated DL & RL, Catalyst-Team, (2019-12-05) 2019年12月5日閲覧。
PiPPy PyTorch Foundation
"TorchScript is executed using an interpreter attached to a JIT-optimizer and compiler." PyTorch. JIT Technical Overview. 2022-03-19閲覧.
"The executor specializes the Graph for the particular set of inputs." PyTorch. JIT Technical Overview. 2022-03-19閲覧.
"It propagates constants, pre-computing as much as possible" PyTorch. JIT Technical Overview. 2022-03-19閲覧.
"Eliminating dead code" PyTorch. JIT Technical Overview. 2022-03-19閲覧.
Fairseq Meta AI
TorchRL PyTorch Foundation