{
"cells": [
{
"cell_type": "markdown",
"metadata": {},
"source": [
"# Multi-Processing-Beispiel\n",
"\n",
"Wir beginnen hier mit Code, der klar und einfach ist und von oben nach unten ausgeführt wird. Er ist einfach zu entwickeln und inkrementell zu testen:"
]
},
{
"cell_type": "code",
"execution_count": 1,
"metadata": {
"execution": {
"iopub.execute_input": "2026-05-21T21:41:20.651686Z",
"iopub.status.busy": "2026-05-21T21:41:20.651442Z",
"iopub.status.idle": "2026-05-21T21:41:21.754611Z",
"shell.execute_reply": "2026-05-21T21:41:21.754059Z",
"shell.execute_reply.started": "2026-05-21T21:41:20.651669Z"
}
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"('https://cusy.io/en', 50276)\n",
"('https://pyviz-tutorial.readthedocs.io/de/latest/', 37736)\n",
"('https://jupyter-tutorial.readthedocs.io/en/latest/', 42963)\n",
"('https://github.com/veit/jupyter-tutorial/', 319332)\n",
"('https://github.com/veit/pyviz-tutorial/', 285369)\n"
]
}
],
"source": [
"from multiprocessing.pool import ThreadPool as Pool\n",
"\n",
"import requests\n",
"\n",
"\n",
"sites = [\n",
" \"https://github.com/veit/jupyter-tutorial/\",\n",
" \"https://jupyter-tutorial.readthedocs.io/en/latest/\",\n",
" \"https://github.com/veit/pyviz-tutorial/\",\n",
" \"https://pyviz-tutorial.readthedocs.io/de/latest/\",\n",
" \"https://cusy.io/en\",\n",
"]\n",
"\n",
"\n",
"def sitesize(url):\n",
" with requests.get(url, timeout=10) as u:\n",
" return url, len(u.content)\n",
"\n",
"\n",
"pool = Pool(10)\n",
"for result in pool.imap_unordered(sitesize, sites):\n",
" print(result)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"\n",
"\n",
"**Hinweis**\n",
" \n",
"Eine gute Entwicklungsstrategie ist die Verwendung von [map](https://docs.python.org/3/library/functions.html#map), um den Code in einem einzelnen Prozess und einem einzelnen Thread zu testen, bevor zu Multi-Processing gewechselt wird.\n",
"
\n",
"\n",
"\n",
"\n",
"**Hinweis**\n",
" \n",
"Um besser einschätzen zu können, wann `ThreadPool` und wann `Pool` verwendet werden sollte, hier einige Faustregeln:\n",
"\n",
"* Für CPU-lastige Jobs sollte `multiprocessing.pool.Pool` verwendet werden. Üblicherweise beginnen wir hier mit der doppelten Anzahl von CPU-Kernen für die Pool-Größe, mindestens jedoch mit 4.\n",
"* Für I/O-lastige Jobs sollte `multiprocessing.pool.ThreadPool` verwendet werden. Üblicherweise beginnen wir hier mit der fünffachen Anzahl von CPU-Kernen für die Pool-Größe.\n",
"* Verwenden wir Python 3 und benötigen kein mit `Pool` identisches Interface, nutzen wir [concurrent.future.Executor](https://docs.python.org/3/library/concurrent.futures.html#concurrent.futures.Executor) statt `multiprocessing.pool.ThreadPool`; er hat ein einfacheres Interface und wurde von Anfang an für Threads konzipiert. Da er Instanzen von `concurrent.futures.Future` zurückgibt, ist er kompatibel zu vielen anderen Bibliotheken, einschließlich `asyncio`.\n",
"* Für CPU- und I/O-lastige Jobs bevorzugen wir `multiprocessing.Pool`, da hierdurch eine bessere Prozess-Isolierung erreicht wird.\n",
"
"
]
},
{
"cell_type": "code",
"execution_count": 2,
"metadata": {
"execution": {
"iopub.execute_input": "2026-05-21T21:41:21.755912Z",
"iopub.status.busy": "2026-05-21T21:41:21.755593Z",
"iopub.status.idle": "2026-05-21T21:41:22.329379Z",
"shell.execute_reply": "2026-05-21T21:41:22.328268Z",
"shell.execute_reply.started": "2026-05-21T21:41:21.755896Z"
}
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"('https://github.com/veit/jupyter-tutorial/', 319332)\n",
"('https://jupyter-tutorial.readthedocs.io/en/latest/', 42963)\n",
"('https://github.com/veit/pyviz-tutorial/', 285364)\n",
"('https://pyviz-tutorial.readthedocs.io/de/latest/', 37736)\n",
"('https://cusy.io/en', 50276)\n"
]
}
],
"source": [
"from multiprocessing.pool import ThreadPool as Pool\n",
"\n",
"\n",
"sites = [\n",
" \"https://github.com/veit/jupyter-tutorial/\",\n",
" \"https://jupyter-tutorial.readthedocs.io/en/latest/\",\n",
" \"https://github.com/veit/pyviz-tutorial/\",\n",
" \"https://pyviz-tutorial.readthedocs.io/de/latest/\",\n",
" \"https://cusy.io/en\",\n",
"]\n",
"\n",
"\n",
"def sitesize(url):\n",
" with requests.get(url, timeout=10) as u:\n",
" return url, len(u.content)\n",
"\n",
"\n",
"for result in map(sitesize, sites):\n",
" print(result)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Was ist parallelisierbar?\n",
"\n",
"### Amdahlsche Gesetz\n",
"\n",
"> Der Geschwindigkeitszuwachs vor allem durch den sequentiellen Anteil des Problems beschränkt, da sich dessen Ausführungszeit durch Parallelisierung nicht verringern lässt. Zudem entstehen durch Parallelisierung zusätzliche Kosten wie etwa für die Kommunikation und die Synchronisierung der Prozesse.\n",
"\n",
"In unserem Beispiel können die folgenden Aufgaben nur seriell abgearbeitet werden:\n",
"\n",
"* UDP DNS request für die URL\n",
"* UDP DNS response\n",
"* Socket vom OS\n",
"* TCP-Connection\n",
"* Senden des HTTP Request für die Root-Ressource\n",
"* Warten auf die TCP Response\n",
"* Zählen der Zeichen auf der Website"
]
},
{
"cell_type": "code",
"execution_count": 3,
"metadata": {
"execution": {
"iopub.execute_input": "2026-05-21T21:41:22.331409Z",
"iopub.status.busy": "2026-05-21T21:41:22.331135Z",
"iopub.status.idle": "2026-05-21T21:41:22.577519Z",
"shell.execute_reply": "2026-05-21T21:41:22.576300Z",
"shell.execute_reply.started": "2026-05-21T21:41:22.331393Z"
}
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"('https://github.com/veit/pyviz-tutorial/', 285363)\n",
"('https://pyviz-tutorial.readthedocs.io/de/latest/', 37736)\n",
"('https://github.com/veit/jupyter-tutorial/', 319332)\n",
"('https://jupyter-tutorial.readthedocs.io/en/latest/', 42963)\n",
"('https://cusy.io/en', 50276)\n"
]
}
],
"source": [
"from multiprocessing.pool import ThreadPool as Pool\n",
"\n",
"\n",
"sites = [\n",
" \"https://github.com/veit/jupyter-tutorial/\",\n",
" \"https://jupyter-tutorial.readthedocs.io/en/latest/\",\n",
" \"https://github.com/veit/pyviz-tutorial/\",\n",
" \"https://pyviz-tutorial.readthedocs.io/de/latest/\",\n",
" \"https://cusy.io/en\",\n",
"]\n",
"\n",
"\n",
"def sitesize(url):\n",
" with requests.get(url, stream=True, timeout=10) as u:\n",
" return url, len(u.content)\n",
"\n",
"\n",
"pool = Pool(4)\n",
"for result in pool.imap_unordered(sitesize, sites):\n",
" print(result)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"\n",
"\n",
"**Hinweis**\n",
" \n",
"[imap_unordered](https://docs.python.org/3/library/multiprocessing.html#multiprocessing.pool.Pool.imap_unordered) wird verwendet, um die Reaktionsfähigkeit zu verbessern. Dies ist jedoch nur möglich, da die Funktion das Argument und das Ergebnis als Tuple zurückgibt."
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Tipps\n",
"\n",
"* Macht nicht zu viele Trips hin und her\n",
"\n",
" Erhaltet ihr zu viele iterierbare Ergebnisse, ist dies ein guter Indikator für zu viele Trips, wie z.B. in\n",
"\n",
" ```python\n",
" >>> def sitesize(url, start):\n",
" ... req = urllib.request.Request()\n",
" ... req.add_header('Range:%d-%d' % (start, start+1000))\n",
" ... u = urllib.request.urlopen(url, req)\n",
" ... block = u.read()\n",
" ... return url, len(block)\n",
" ```\n",
"\n",
"* Macht auf jedem Trip relevante Fortschritte\n",
"\n",
" Sobald ihr den Prozess erhaltet, solltet ihr deutliche Fortschritte erzielen und euch nicht verzetteln. Das folgende Beispiel verdeutlicht zu kleine Zwischenschritte:\n",
"\n",
" ```python\n",
" >>> def sitesize(url, results):\n",
" ... with requests.get(url, stream=True) as u:\n",
" ... while True:\n",
" ... line = u.iter_lines()\n",
" ... results.put((url, len(line)))\n",
" ```\n",
"\n",
"* Sendet und empfangt nicht zu viele Daten\n",
"\n",
" Das folgende Beispiel erhöht unnötig die Datenmenge:\n",
"\n",
" ```python\n",
" >>> def sitesize(url):\n",
" ... with requests.get(url) as u:\n",
" ... return url, u.content\n",
" ```"
]
}
],
"metadata": {
"kernelspec": {
"display_name": "Python 3.11 Kernel",
"language": "python",
"name": "python311"
},
"language_info": {
"codemirror_mode": {
"name": "ipython",
"version": 3
},
"file_extension": ".py",
"mimetype": "text/x-python",
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
"version": "3.11.14"
},
"latex_envs": {
"LaTeX_envs_menu_present": true,
"autoclose": false,
"autocomplete": true,
"bibliofile": "biblio.bib",
"cite_by": "apalike",
"current_citInitial": 1,
"eqLabelWithNumbers": true,
"eqNumInitial": 1,
"hotkeys": {
"equation": "Ctrl-E",
"itemize": "Ctrl-I"
},
"labels_anchors": false,
"latex_user_defs": false,
"report_style_numbering": false,
"user_envs_cfg": false
},
"varInspector": {
"cols": {
"lenName": 16,
"lenType": 16,
"lenVar": 40
},
"kernels_config": {
"python": {
"delete_cmd_postfix": "",
"delete_cmd_prefix": "del ",
"library": "var_list.py",
"varRefreshCmd": "print(var_dic_list())"
},
"r": {
"delete_cmd_postfix": ") ",
"delete_cmd_prefix": "rm(",
"library": "var_list.r",
"varRefreshCmd": "cat(var_dic_list()) "
}
},
"types_to_exclude": [
"module",
"function",
"builtin_function_or_method",
"instance",
"_Feature"
],
"window_display": false
},
"widgets": {
"application/vnd.jupyter.widget-state+json": {
"state": {},
"version_major": 2,
"version_minor": 0
}
}
},
"nbformat": 4,
"nbformat_minor": 4
}