サブロウ丸

サブロウ丸

主にプログラミングと数学

論文サーベイ

サーベイ: Automatic Graph Partitioning for Very Large-scale Deep Learning

Tanaka, Masahiro, et al. "Automatic graph partitioning for very large-scale deep learning." 2021 IEEE International Parallel and Distributed Processing Symposium (IPDPS). IEEE, 2021. @inproceedings{tanaka2021automatic, title={Automatic gra…

サーベイ: Supporting Very Large Models using Automatic Dataflow Graph Partitioning

Wang, Minjie, Chien-chin Huang, and Jinyang Li. "Supporting very large models using automatic dataflow graph partitioning." Proceedings of the Fourteenth EuroSys Conference 2019. 2019. @inproceedings{wang2019supporting, title={Supporting v…

サーベイ: Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelism

@article{shoeybi2019megatron, title={Megatron-lm: Training multi-billion parameter language models using model parallelism}, author={Shoeybi, Mohammad and Patwary, Mostofa and Puri, Raul and LeGresley, Patrick and Casper, Jared and Catanza…

サーベイ: ZeRO-Offload: Democratizing Billion-Scale Model Training

@inproceedings{ren2021zero, title={$\{$ZeRO-Offload$\}$: Democratizing $\{$Billion-Scale$\}$ Model Training}, author={Ren, Jie and Rajbhandari, Samyam and Aminabadi, Reza Yazdani and Ruwase, Olatunji and Yang, Shuangyan and Zhang, Minjia a…

サーベイ: ZeRO: Memory Optimizations Toward Training Trillion Parameter Models

@inproceedings{rajbhandari2020zero, title={Zero: Memory optimizations toward training trillion parameter models}, author={Rajbhandari, Samyam and Rasley, Jeff and Ruwase, Olatunji and He, Yuxiong}, booktitle={SC20: International Conference…

サーベイ: Training Deep Nets with Sublinear Memory Cost

Chen, Tianqi, et al. "Training deep nets with sublinear memory cost." arXiv preprint arXiv:1604.06174 (2016). @article{chen2016training, title={Training deep nets with sublinear memory cost}, author={Chen, Tianqi and Xu, Bing and Zhang, Ch…

サーベイ: GPUメモリ管理の実行時最適化による大規模深層学習の高速化

@article{伊藤祐貴2018gpu, title={GPU メモリ管理の実行時最適化による大規模深層学習の高速化}, author={伊藤祐貴 and 今井晴基 and 根岸康 and 河内谷清久仁 and 松宮遼 and 遠藤敏夫 and others}, journal={研究報告ハイパフォーマンスコンピューティン…

サーベイ: Efficient Large-Scale Language Model Training on GPU Clusters Using Megatron-LM

https://dl.acm.org/doi/10.1145/3458817.3476209 @inproceedings{10.1145/3458817.3476209, author = {Narayanan, Deepak and Shoeybi, Mohammad and Casper, Jared and LeGresley, Patrick and Patwary, Mostofa and Korthikanti, Vijay and Vainbrand, Dm…

サーベイ: Mesh-tensorflow:Deep learning for supercomputers

@article{shazeer2018mesh, title={Mesh-tensorflow: Deep learning for supercomputers}, author={Shazeer, Noam and Cheng, Youlong and Parmar, Niki and Tran, Dustin and Vaswani, Ashish and Koanantakool, Penporn and Hawkins, Peter and Lee, Hyouk…

サーベイ: PipeDream: Generalized Pipeline Parallelism for DNN Training

https://dl.acm.org/doi/abs/10.1145/3341301.3359646?casa_token=L-sKQKrRoE4AAAAA%3AYKo9NPdnPyG6IouMN5jfTHTCYFAGORDxen32GKAteeSG-ROhqx_OX-hVOfuyHiVBXLLJH0RPujhFPEk @inproceedings{narayanan2019pipedream, title={PipeDream: generalized pipeline …

サーベイ: Gpipe: Efficient training of giant neural networks using pipeline parallelism

@article{huang2019gpipe, title={Gpipe: Efficient training of giant neural networks using pipeline parallelism}, author={Huang, Yanping and Cheng, Youlong and Bapna, Ankur and Firat, Orhan and Chen, Dehao and Chen, Mia and Lee, HyoukJoong a…

サーベイ: 分散深層学習

深層学習において、学習データと学習モデルの巨大化が最新のトレンドになっています。 そこで学習時間の削減のために複数のマシンを用いてモデルを訓練する試みが行われており、 分散深層学習(distributed deep learning)などという呼ばれ方で一つの分野にな…

サーベイ: From sequential algorithm selection to parallel portfolio selection

Lindauer, Marius, Holger Hoos, and Frank Hutter. "From sequential algorithm selection to parallel portfolio selection." International Conference on Learning and Intelligent Optimization. Springer, Cham, 2015. ポートフォリオ最適化とは(1) 資…

サーベイ: A survey on single crane scheduling in automated storage/retrieval systems

Nils Boysen and Konrad Stephan. “A survey on single crane scheduling in automated stor- age/retrieval systems”. In: European Journal of Operational Research 254.3 (2016), pp. 691– 704. issn: 0377-2217. doi: https://doi.org/10.1016/j.ejor.2…

サーベイ; Simulation study of an automated storage/retrieval system

Jeroen P. van den Berg and A.J.R.M. Gademann. “Simulation study of an automated storage/retrieval system”. In: International Journal of Production Research 38.6 (2000), pp. 1339–1356. doi: 10.1080/002075400188889. eprint: https://doi.org/1…

サーベイ; The quantum or not to quantum: towards algorithm selection in near-term quantum optimization

2020年の論文, "MaxCut algorithm selection"の検索でヒット; Moussa, Charles, Henri Calandra, and Vedran Dunjko. "To quantum or not to quantum: towards algorithm selection in near-term quantum optimization." Quantum Science and Technology 5.4…