You probably tried many times to execute a script on multiple files in parallel, or to download multiple files at the same time, only to find out that you’ve exhausted your machine’s resources or that some jobs have failed and needs to be retried. GNU Parallel is a tool for executing jobs in parallel using one or more computers. In this talk we will see how GNU Parallel makes it easy to distribute and monitor tasks, independently of the programming language.
Commands used in this session:
Dataset of URLs used as an example: