{"id":2139,"date":"2023-12-16T23:53:30","date_gmt":"2023-12-16T15:53:30","guid":{"rendered":"https:\/\/pjq.me\/?p=2139"},"modified":"2023-12-31T19:08:21","modified_gmt":"2023-12-31T11:08:21","slug":"how-to-run-llm-locally","status":"publish","type":"post","link":"https:\/\/pjq.me\/?p=2139","title":{"rendered":"How to run LLM locally"},"content":{"rendered":"\n<p class=\"wp-block-paragraph\">There are so many large language models that you can run it locally, e.g. llama, mixtral.<\/p>\n\n\n\n<div id=\"ez-toc-container\" class=\"ez-toc-v2_0_85 counter-hierarchy ez-toc-counter ez-toc-grey ez-toc-container-direction\">\n<div class=\"ez-toc-title-container\">\n<p class=\"ez-toc-title\" style=\"cursor:inherit\">Table of Contents<\/p>\n<span class=\"ez-toc-title-toggle\"><a href=\"#\" class=\"ez-toc-pull-right ez-toc-btn ez-toc-btn-xs ez-toc-btn-default ez-toc-toggle\" aria-label=\"Toggle Table of Content\"><span class=\"ez-toc-js-icon-con\"><span class=\"\"><span class=\"eztoc-hide\" style=\"display:none;\">Toggle<\/span><span class=\"ez-toc-icon-toggle-span\"><svg style=\"fill: #999;color:#999\" xmlns=\"http:\/\/www.w3.org\/2000\/svg\" class=\"list-377408\" width=\"20px\" height=\"20px\" viewBox=\"0 0 24 24\" fill=\"none\"><path d=\"M6 6H4v2h2V6zm14 0H8v2h12V6zM4 11h2v2H4v-2zm16 0H8v2h12v-2zM4 16h2v2H4v-2zm16 0H8v2h12v-2z\" fill=\"currentColor\"><\/path><\/svg><svg style=\"fill: #999;color:#999\" class=\"arrow-unsorted-368013\" xmlns=\"http:\/\/www.w3.org\/2000\/svg\" width=\"10px\" height=\"10px\" viewBox=\"0 0 24 24\" version=\"1.2\" baseProfile=\"tiny\"><path d=\"M18.2 9.3l-6.2-6.3-6.2 6.3c-.2.2-.3.4-.3.7s.1.5.3.7c.2.2.4.3.7.3h11c.3 0 .5-.1.7-.3.2-.2.3-.5.3-.7s-.1-.5-.3-.7zM5.8 14.7l6.2 6.3 6.2-6.3c.2-.2.3-.5.3-.7s-.1-.5-.3-.7c-.2-.2-.4-.3-.7-.3h-11c-.3 0-.5.1-.7.3-.2.2-.3.5-.3.7s.1.5.3.7z\"\/><\/svg><\/span><\/span><\/span><\/a><\/span><\/div>\n<nav><ul class='ez-toc-list ez-toc-list-level-1 ' ><li class='ez-toc-page-1 ez-toc-heading-level-2'><a class=\"ez-toc-link ez-toc-heading-1\" href=\"https:\/\/pjq.me\/?p=2139\/#llamacpp\" >llama.cpp<\/a><\/li><li class='ez-toc-page-1 ez-toc-heading-level-2'><a class=\"ez-toc-link ez-toc-heading-2\" href=\"https:\/\/pjq.me\/?p=2139\/#ollama\" >ollama<\/a><\/li><li class='ez-toc-page-1 ez-toc-heading-level-2'><a class=\"ez-toc-link ez-toc-heading-3\" href=\"https:\/\/pjq.me\/?p=2139\/#llamafile\" >llamafile<\/a><\/li><\/ul><\/nav><\/div>\n<h2 class=\"wp-block-heading\"><span class=\"ez-toc-section\" id=\"llamacpp\"><\/span>llama.cpp<span class=\"ez-toc-section-end\"><\/span><\/h2>\n\n\n\n<p class=\"wp-block-paragraph\">The most popular open source LLM framework that support run many LLM locally.<\/p>\n\n\n\n<ul class=\"wp-block-list\">\n<li><a href=\"https:\/\/github.com\/ggerganov\/llama.cpp\/\">https:\/\/github.com\/ggerganov\/llama.cpp\/<\/a><\/li>\n<\/ul>\n\n\n\n<pre class=\"wp-block-code\"><code>git clone https:\/\/github.com\/ggerganov\/llama.cpp.git\ncd llama.cpp\nmake\n.\/examples\/chat-13B.sh<\/code><\/pre>\n\n\n\n<p class=\"wp-block-paragraph\">Before you can run the examples, you need download the models first.<\/p>\n\n\n\n<h2 class=\"wp-block-heading\"><span class=\"ez-toc-section\" id=\"ollama\"><\/span>ollama<span class=\"ez-toc-section-end\"><\/span><\/h2>\n\n\n\n<ul class=\"wp-block-list\">\n<li><a href=\"https:\/\/ollama.ai\/library\/dolphin-mixtral\/tags\">https:\/\/ollama.ai\/library\/dolphin-mixtral\/tags<\/a><\/li>\n<\/ul>\n\n\n\n<p class=\"wp-block-paragraph\">Install the ollama first, then run it<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>ollama run dolphin-mixtral\npulling manifest\npulling bdb11b0699e0...  84% \u2595\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588        \u258f  22 GB\/ 26 GB  6.0 MB\/s  11m<\/code><\/pre>\n\n\n\n<pre class=\"wp-block-code\"><code>ollama -h\nLarge language model runner\n\nUsage:\n  ollama &#91;flags]\n  ollama &#91;command]\n\nAvailable Commands:\n  serve       Start ollama\n  create      Create a model from a Modelfile\n  show        Show information for a model\n  run         Run a model\n  pull        Pull a model from a registry\n  push        Push a model to a registry\n  list        List models\n  cp          Copy a model\n  rm          Remove a model\n  help        Help about any command<\/code><\/pre>\n\n\n\n<h2 class=\"wp-block-heading\"><span class=\"ez-toc-section\" id=\"llamafile\"><\/span>llamafile<span class=\"ez-toc-section-end\"><\/span><\/h2>\n\n\n\n<p class=\"wp-block-paragraph\">One file that include everything, so just download the llamafile, and run it.<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code> wget -cS https:\/\/huggingface.co\/jartine\/mistral-7b.llamafile\/resolve\/main\/mistral-7b-instruct-v0.1-Q4_K_M-server.llamafile\nchmod +x mistral-7b-instruct-v0.1-Q4_K_M-server.llamafile\n.\/mistral-7b-instruct-v0.1-Q4_K_M-server.llamafile<\/code><\/pre>\n\n\n\n<div class=\"wp-block-media-text is-stacked-on-mobile\"><figure class=\"wp-block-media-text__media\"><img loading=\"lazy\" decoding=\"async\" width=\"758\" height=\"1024\" src=\"https:\/\/pjq.me\/wp-content\/uploads\/2023\/12\/image-4-758x1024.png\" alt=\"\" class=\"wp-image-2140 size-full\" srcset=\"https:\/\/pjq.me\/wp-content\/uploads\/2023\/12\/image-4-758x1024.png 758w, https:\/\/pjq.me\/wp-content\/uploads\/2023\/12\/image-4-222x300.png 222w, https:\/\/pjq.me\/wp-content\/uploads\/2023\/12\/image-4-768x1038.png 768w, https:\/\/pjq.me\/wp-content\/uploads\/2023\/12\/image-4-1137x1536.png 1137w, https:\/\/pjq.me\/wp-content\/uploads\/2023\/12\/image-4.png 1360w\" sizes=\"auto, (max-width: 758px) 100vw, 758px\" \/><\/figure><div class=\"wp-block-media-text__content\">\n<p class=\"wp-block-paragraph\"><\/p>\n<\/div><\/div>\n","protected":false},"excerpt":{"rendered":"<p>There are so many large language models that you can run it locally, e.g. llama, mixtral. llama.cpp The most popular open source LLM framework that support run many LLM locally. Before you can run the examples, you need download the<\/p>\n","protected":false},"author":1,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[169],"tags":[202,229],"class_list":["post-2139","post","type-post","status-publish","format-standard","hentry","category-tech","tag-ai","tag-llm"],"yoast_head":"<!-- This site is optimized with the Yoast SEO plugin v28.0 - https:\/\/yoast.com\/product\/yoast-seo-wordpress\/ -->\n<title>How to run LLM locally - Jianqing&#039;s Blog<\/title>\n<meta name=\"robots\" content=\"index, follow, max-snippet:-1, max-image-preview:large, max-video-preview:-1\" \/>\n<link rel=\"canonical\" href=\"https:\/\/pjq.me\/?p=2139\" \/>\n<meta property=\"og:locale\" content=\"en_US\" \/>\n<meta property=\"og:type\" content=\"article\" \/>\n<meta property=\"og:title\" content=\"How to run LLM locally - Jianqing&#039;s Blog\" \/>\n<meta property=\"og:description\" content=\"There are so many large language models that you can run it locally, e.g. llama, mixtral. llama.cpp The most popular open source LLM framework that support run many LLM locally. Before you can run the examples, you need download the\" \/>\n<meta property=\"og:url\" content=\"https:\/\/pjq.me\/?p=2139\" \/>\n<meta property=\"og:site_name\" content=\"Jianqing&#039;s Blog\" \/>\n<meta property=\"article:published_time\" content=\"2023-12-16T15:53:30+00:00\" \/>\n<meta property=\"article:modified_time\" content=\"2023-12-31T11:08:21+00:00\" \/>\n<meta property=\"og:image\" content=\"https:\/\/pjq.me\/wp-content\/uploads\/2023\/12\/image-4-758x1024.png\" \/>\n<meta name=\"author\" content=\"pengjianqing\" \/>\n<meta name=\"twitter:card\" content=\"summary_large_image\" \/>\n<meta name=\"twitter:label1\" content=\"Written by\" \/>\n\t<meta name=\"twitter:data1\" content=\"pengjianqing\" \/>\n\t<meta name=\"twitter:label2\" content=\"Est. reading time\" \/>\n\t<meta name=\"twitter:data2\" content=\"1 minute\" \/>\n<script type=\"application\/ld+json\" class=\"yoast-schema-graph\">{\"@context\":\"https:\\\/\\\/schema.org\",\"@graph\":[{\"@type\":\"Article\",\"@id\":\"https:\\\/\\\/pjq.me\\\/?p=2139#article\",\"isPartOf\":{\"@id\":\"https:\\\/\\\/pjq.me\\\/?p=2139\"},\"author\":{\"name\":\"pengjianqing\",\"@id\":\"https:\\\/\\\/pjq.me\\\/#\\\/schema\\\/person\\\/0eb1e72d1e69fbbd9b5c0bfd8e2aae60\"},\"headline\":\"How to run LLM locally\",\"datePublished\":\"2023-12-16T15:53:30+00:00\",\"dateModified\":\"2023-12-31T11:08:21+00:00\",\"mainEntityOfPage\":{\"@id\":\"https:\\\/\\\/pjq.me\\\/?p=2139\"},\"wordCount\":83,\"commentCount\":0,\"publisher\":{\"@id\":\"https:\\\/\\\/pjq.me\\\/#\\\/schema\\\/person\\\/0eb1e72d1e69fbbd9b5c0bfd8e2aae60\"},\"image\":{\"@id\":\"https:\\\/\\\/pjq.me\\\/?p=2139#primaryimage\"},\"thumbnailUrl\":\"https:\\\/\\\/pjq.me\\\/wp-content\\\/uploads\\\/2023\\\/12\\\/image-4-758x1024.png\",\"keywords\":[\"AI\",\"LLM\"],\"articleSection\":[\"Tech\"],\"inLanguage\":\"en-US\",\"potentialAction\":[{\"@type\":\"CommentAction\",\"name\":\"Comment\",\"target\":[\"https:\\\/\\\/pjq.me\\\/?p=2139#respond\"]}]},{\"@type\":\"WebPage\",\"@id\":\"https:\\\/\\\/pjq.me\\\/?p=2139\",\"url\":\"https:\\\/\\\/pjq.me\\\/?p=2139\",\"name\":\"How to run LLM locally - Jianqing&#039;s Blog\",\"isPartOf\":{\"@id\":\"https:\\\/\\\/pjq.me\\\/#website\"},\"primaryImageOfPage\":{\"@id\":\"https:\\\/\\\/pjq.me\\\/?p=2139#primaryimage\"},\"image\":{\"@id\":\"https:\\\/\\\/pjq.me\\\/?p=2139#primaryimage\"},\"thumbnailUrl\":\"https:\\\/\\\/pjq.me\\\/wp-content\\\/uploads\\\/2023\\\/12\\\/image-4-758x1024.png\",\"datePublished\":\"2023-12-16T15:53:30+00:00\",\"dateModified\":\"2023-12-31T11:08:21+00:00\",\"breadcrumb\":{\"@id\":\"https:\\\/\\\/pjq.me\\\/?p=2139#breadcrumb\"},\"inLanguage\":\"en-US\",\"potentialAction\":[{\"@type\":\"ReadAction\",\"target\":[\"https:\\\/\\\/pjq.me\\\/?p=2139\"]}]},{\"@type\":\"ImageObject\",\"inLanguage\":\"en-US\",\"@id\":\"https:\\\/\\\/pjq.me\\\/?p=2139#primaryimage\",\"url\":\"https:\\\/\\\/pjq.me\\\/wp-content\\\/uploads\\\/2023\\\/12\\\/image-4.png\",\"contentUrl\":\"https:\\\/\\\/pjq.me\\\/wp-content\\\/uploads\\\/2023\\\/12\\\/image-4.png\",\"width\":1360,\"height\":1838},{\"@type\":\"BreadcrumbList\",\"@id\":\"https:\\\/\\\/pjq.me\\\/?p=2139#breadcrumb\",\"itemListElement\":[{\"@type\":\"ListItem\",\"position\":1,\"name\":\"Home\",\"item\":\"https:\\\/\\\/pjq.me\\\/\"},{\"@type\":\"ListItem\",\"position\":2,\"name\":\"How to run LLM locally\"}]},{\"@type\":\"WebSite\",\"@id\":\"https:\\\/\\\/pjq.me\\\/#website\",\"url\":\"https:\\\/\\\/pjq.me\\\/\",\"name\":\"Jianqing&#039;s Blog\",\"description\":\"Thoughts and Future\",\"publisher\":{\"@id\":\"https:\\\/\\\/pjq.me\\\/#\\\/schema\\\/person\\\/0eb1e72d1e69fbbd9b5c0bfd8e2aae60\"},\"potentialAction\":[{\"@type\":\"SearchAction\",\"target\":{\"@type\":\"EntryPoint\",\"urlTemplate\":\"https:\\\/\\\/pjq.me\\\/?s={search_term_string}\"},\"query-input\":{\"@type\":\"PropertyValueSpecification\",\"valueRequired\":true,\"valueName\":\"search_term_string\"}}],\"inLanguage\":\"en-US\"},{\"@type\":[\"Person\",\"Organization\"],\"@id\":\"https:\\\/\\\/pjq.me\\\/#\\\/schema\\\/person\\\/0eb1e72d1e69fbbd9b5c0bfd8e2aae60\",\"name\":\"pengjianqing\",\"image\":{\"@type\":\"ImageObject\",\"inLanguage\":\"en-US\",\"@id\":\"https:\\\/\\\/pjq.me\\\/wp-content\\\/uploads\\\/2021\\\/12\\\/Screen-Shot-2021-12-02-at-6.10.58-PM.png\",\"url\":\"https:\\\/\\\/pjq.me\\\/wp-content\\\/uploads\\\/2021\\\/12\\\/Screen-Shot-2021-12-02-at-6.10.58-PM.png\",\"contentUrl\":\"https:\\\/\\\/pjq.me\\\/wp-content\\\/uploads\\\/2021\\\/12\\\/Screen-Shot-2021-12-02-at-6.10.58-PM.png\",\"width\":460,\"height\":752,\"caption\":\"pengjianqing\"},\"logo\":{\"@id\":\"https:\\\/\\\/pjq.me\\\/wp-content\\\/uploads\\\/2021\\\/12\\\/Screen-Shot-2021-12-02-at-6.10.58-PM.png\"},\"url\":\"https:\\\/\\\/pjq.me\\\/?author=1\"}]}<\/script>\n<!-- \/ Yoast SEO plugin. -->","yoast_head_json":{"title":"How to run LLM locally - Jianqing&#039;s Blog","robots":{"index":"index","follow":"follow","max-snippet":"max-snippet:-1","max-image-preview":"max-image-preview:large","max-video-preview":"max-video-preview:-1"},"canonical":"https:\/\/pjq.me\/?p=2139","og_locale":"en_US","og_type":"article","og_title":"How to run LLM locally - Jianqing&#039;s Blog","og_description":"There are so many large language models that you can run it locally, e.g. llama, mixtral. llama.cpp The most popular open source LLM framework that support run many LLM locally. Before you can run the examples, you need download the","og_url":"https:\/\/pjq.me\/?p=2139","og_site_name":"Jianqing&#039;s Blog","article_published_time":"2023-12-16T15:53:30+00:00","article_modified_time":"2023-12-31T11:08:21+00:00","og_image":[{"url":"https:\/\/pjq.me\/wp-content\/uploads\/2023\/12\/image-4-758x1024.png","type":"","width":"","height":""}],"author":"pengjianqing","twitter_card":"summary_large_image","twitter_misc":{"Written by":"pengjianqing","Est. reading time":"1 minute"},"schema":{"@context":"https:\/\/schema.org","@graph":[{"@type":"Article","@id":"https:\/\/pjq.me\/?p=2139#article","isPartOf":{"@id":"https:\/\/pjq.me\/?p=2139"},"author":{"name":"pengjianqing","@id":"https:\/\/pjq.me\/#\/schema\/person\/0eb1e72d1e69fbbd9b5c0bfd8e2aae60"},"headline":"How to run LLM locally","datePublished":"2023-12-16T15:53:30+00:00","dateModified":"2023-12-31T11:08:21+00:00","mainEntityOfPage":{"@id":"https:\/\/pjq.me\/?p=2139"},"wordCount":83,"commentCount":0,"publisher":{"@id":"https:\/\/pjq.me\/#\/schema\/person\/0eb1e72d1e69fbbd9b5c0bfd8e2aae60"},"image":{"@id":"https:\/\/pjq.me\/?p=2139#primaryimage"},"thumbnailUrl":"https:\/\/pjq.me\/wp-content\/uploads\/2023\/12\/image-4-758x1024.png","keywords":["AI","LLM"],"articleSection":["Tech"],"inLanguage":"en-US","potentialAction":[{"@type":"CommentAction","name":"Comment","target":["https:\/\/pjq.me\/?p=2139#respond"]}]},{"@type":"WebPage","@id":"https:\/\/pjq.me\/?p=2139","url":"https:\/\/pjq.me\/?p=2139","name":"How to run LLM locally - Jianqing&#039;s Blog","isPartOf":{"@id":"https:\/\/pjq.me\/#website"},"primaryImageOfPage":{"@id":"https:\/\/pjq.me\/?p=2139#primaryimage"},"image":{"@id":"https:\/\/pjq.me\/?p=2139#primaryimage"},"thumbnailUrl":"https:\/\/pjq.me\/wp-content\/uploads\/2023\/12\/image-4-758x1024.png","datePublished":"2023-12-16T15:53:30+00:00","dateModified":"2023-12-31T11:08:21+00:00","breadcrumb":{"@id":"https:\/\/pjq.me\/?p=2139#breadcrumb"},"inLanguage":"en-US","potentialAction":[{"@type":"ReadAction","target":["https:\/\/pjq.me\/?p=2139"]}]},{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/pjq.me\/?p=2139#primaryimage","url":"https:\/\/pjq.me\/wp-content\/uploads\/2023\/12\/image-4.png","contentUrl":"https:\/\/pjq.me\/wp-content\/uploads\/2023\/12\/image-4.png","width":1360,"height":1838},{"@type":"BreadcrumbList","@id":"https:\/\/pjq.me\/?p=2139#breadcrumb","itemListElement":[{"@type":"ListItem","position":1,"name":"Home","item":"https:\/\/pjq.me\/"},{"@type":"ListItem","position":2,"name":"How to run LLM locally"}]},{"@type":"WebSite","@id":"https:\/\/pjq.me\/#website","url":"https:\/\/pjq.me\/","name":"Jianqing&#039;s Blog","description":"Thoughts and Future","publisher":{"@id":"https:\/\/pjq.me\/#\/schema\/person\/0eb1e72d1e69fbbd9b5c0bfd8e2aae60"},"potentialAction":[{"@type":"SearchAction","target":{"@type":"EntryPoint","urlTemplate":"https:\/\/pjq.me\/?s={search_term_string}"},"query-input":{"@type":"PropertyValueSpecification","valueRequired":true,"valueName":"search_term_string"}}],"inLanguage":"en-US"},{"@type":["Person","Organization"],"@id":"https:\/\/pjq.me\/#\/schema\/person\/0eb1e72d1e69fbbd9b5c0bfd8e2aae60","name":"pengjianqing","image":{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/pjq.me\/wp-content\/uploads\/2021\/12\/Screen-Shot-2021-12-02-at-6.10.58-PM.png","url":"https:\/\/pjq.me\/wp-content\/uploads\/2021\/12\/Screen-Shot-2021-12-02-at-6.10.58-PM.png","contentUrl":"https:\/\/pjq.me\/wp-content\/uploads\/2021\/12\/Screen-Shot-2021-12-02-at-6.10.58-PM.png","width":460,"height":752,"caption":"pengjianqing"},"logo":{"@id":"https:\/\/pjq.me\/wp-content\/uploads\/2021\/12\/Screen-Shot-2021-12-02-at-6.10.58-PM.png"},"url":"https:\/\/pjq.me\/?author=1"}]}},"views":2024,"_links":{"self":[{"href":"https:\/\/pjq.me\/index.php?rest_route=\/wp\/v2\/posts\/2139","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/pjq.me\/index.php?rest_route=\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/pjq.me\/index.php?rest_route=\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/pjq.me\/index.php?rest_route=\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/pjq.me\/index.php?rest_route=%2Fwp%2Fv2%2Fcomments&post=2139"}],"version-history":[{"count":2,"href":"https:\/\/pjq.me\/index.php?rest_route=\/wp\/v2\/posts\/2139\/revisions"}],"predecessor-version":[{"id":2142,"href":"https:\/\/pjq.me\/index.php?rest_route=\/wp\/v2\/posts\/2139\/revisions\/2142"}],"wp:attachment":[{"href":"https:\/\/pjq.me\/index.php?rest_route=%2Fwp%2Fv2%2Fmedia&parent=2139"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/pjq.me\/index.php?rest_route=%2Fwp%2Fv2%2Fcategories&post=2139"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/pjq.me\/index.php?rest_route=%2Fwp%2Fv2%2Ftags&post=2139"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}