ray-crash-course/05-Running-Ray-Clusters.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Ray Crash Course - Ray Clusters and the Ray CLI\n",
    "\n",
    "© 2019-2022, Anyscale. All Rights Reserved\n",
    "\n",
    "![Anyscale Academy](../images/AnyscaleAcademyLogo.png)\n",
    "\n",
    "In the previous lessons, we let `ray.init()` start a mini-cluster on your laptop or connect to the running Ray cluster in the Anyscale hosted platform. This lesson discusses using the Ray CLI command `ray` to create and manage Ray clusters. We won't cover all the subcommands `ray` supports. Try `ray --help` and see the [Ray CLI documentation](https://docs.ray.io/en/latest/package-ref.html#the-ray-command-line-api) for more details."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "> **Tip:** If any of the CLI commands used here print a lot of output, right click on the output and select _Enable Scrolling for Outputs_.\n",
    "\n",
    "> **Notes:** \n",
    ">\n",
    "> 1. The Anyscale hosted platform has its own CLI command, `anyscale`, which integrates the `ray` CLI and provides other capabilities for managing and running Ray projects and sessions, including automated cluster integration, synchronization of code to your local development environment, etc. Further information on this service will be available soon. [Contact us](mailto:academy@anyscale.com) for details.\n",
    "> 2. Ray can now be used with [Docker](https://www.docker.com/). You can find the published Docker images [here](https://hub.docker.com/search?q=Rayproject&type=image). For more details, see the documentation [here](https://docs.ray.io/en/latest/installation.html?highlight=docker#launch-ray-in-docker) and [here](https://docs.ray.io/en/latest/cluster/cloud.html?highlight=docker#common-cluster-configurations)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## ray --help\n",
    "\n",
    "The typical `help` information is available with `--help` or with no arguments:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Usage: ray [OPTIONS] COMMAND [ARGS]...\n",
      "\n",
      "Options:\n",
      "  --logging-level TEXT   The logging level threshold, choices=['debug',\n",
      "                         'info', 'warning', 'error', 'critical'],\n",
      "                         default='info'\n",
      "  --logging-format TEXT  The logging format. default='%(asctime)s\n",
      "                         %(levelname)s %(filename)s:%(lineno)s -- %(message)s'\n",
      "  --version              Show the version and exit.\n",
      "  --help                 Show this message and exit.\n",
      "\n",
      "Commands:\n",
      "  attach           Create or attach to a SSH session to a Ray cluster.\n",
      "  cluster-dump     Get log data from one or more nodes.\n",
      "  cpp              Show the cpp library path and generate the bazel...\n",
      "  dashboard        Port-forward a Ray cluster's dashboard to the local...\n",
      "  debug            Show all active breakpoints and exceptions in the Ray...\n",
      "  down             Tear down a Ray cluster.\n",
      "  exec             Execute a command via SSH on a Ray cluster.\n",
      "  get-head-ip      Return the head node IP of a Ray cluster.\n",
      "  get-worker-ips   Return the list of worker IPs of a Ray cluster.\n",
      "  install-nightly  Install the latest wheels for Ray.\n",
      "  memory           Print object references held in a Ray cluster.\n",
      "  microbenchmark   Run a local Ray microbenchmark on the current machine.\n",
      "  monitor          Tails the autoscaler logs of a Ray cluster.\n",
      "  rsync-down       Download specific files from a Ray cluster.\n",
      "  rsync-up         Upload specific files to a Ray cluster.\n",
      "  stack            Take a stack dump of all Python workers on the local...\n",
      "  start            Start Ray processes manually on the local machine.\n",
      "  status           Print cluster status, including autoscaling info.\n",
      "  stop             Stop Ray processes manually on the local machine.\n",
      "  submit           Uploads and runs a script on the specified cluster.\n",
      "  timeline         Take a Chrome tracing timeline for a Ray cluster.\n",
      "  up               Create or update a Ray cluster.\n",
      "\u001b[0m"
     ]
    }
   ],
   "source": [
    "!ray --help"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Some of these commands are aliases, e.g., `down` and `teardown`, `get-head-ip` and `get_head_ip`, etc. `kill-random-node` looks strange, but it is useful for [Chaos Engineering](https://en.wikipedia.org/wiki/Chaos_engineering) purposes. \n",
    "\n",
    "For more details on a particular command, use `ray <command> --help`:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Usage: ray start [OPTIONS]\n",
      "\n",
      "  Start Ray processes manually on the local machine.\n",
      "\n",
      "Options:\n",
      "  --node-ip-address TEXT          the IP address of this node\n",
      "  --address TEXT                  the address to use for Ray\n",
      "  --port INTEGER                  the port of the head ray process. If not\n",
      "                                  provided, defaults to 6379; if port is set\n",
      "                                  to 0, we will allocate an available port.\n",
      "  --object-manager-port INTEGER   the port to use for starting the object\n",
      "                                  manager\n",
      "  --node-manager-port INTEGER     the port to use for starting the node\n",
      "                                  manager\n",
      "  --gcs-server-port INTEGER       Port number for the GCS server.\n",
      "  --min-worker-port INTEGER       the lowest port number that workers will\n",
      "                                  bind on. If not set, random ports will be\n",
      "                                  chosen.\n",
      "  --max-worker-port INTEGER       the highest port number that workers will\n",
      "                                  bind on. If set, '--min-worker-port' must\n",
      "                                  also be set.\n",
      "  --worker-port-list TEXT         a comma-separated list of open ports for\n",
      "                                  workers to bind on. Overrides '--min-worker-\n",
      "                                  port' and '--max-worker-port'.\n",
      "  --ray-client-server-port INTEGER\n",
      "                                  the port number the ray client server will\n",
      "                                  bind on. If not set, the ray client server\n",
      "                                  will not be started.\n",
      "  --object-store-memory INTEGER   The amount of memory (in bytes) to start the\n",
      "                                  object store with. By default, this is\n",
      "                                  capped at 20GB but can be set higher.\n",
      "  --num-cpus INTEGER              the number of CPUs on this node\n",
      "  --num-gpus INTEGER              the number of GPUs on this node\n",
      "  --resources TEXT                a JSON serialized dictionary mapping\n",
      "                                  resource name to resource quantity\n",
      "  --head                          provide this argument for the head node\n",
      "  --include-dashboard BOOLEAN     provide this argument to start the Ray\n",
      "                                  dashboard GUI\n",
      "  --dashboard-host TEXT           the host to bind the dashboard server to,\n",
      "                                  either localhost (127.0.0.1) or 0.0.0.0\n",
      "                                  (available from all interfaces). By default,\n",
      "                                  thisis localhost.\n",
      "  --dashboard-port INTEGER        the port to bind the dashboard server to--\n",
      "                                  defaults to 8265\n",
      "  --block                         provide this argument to block forever in\n",
      "                                  this command\n",
      "  --plasma-directory TEXT         object store directory for memory mapped\n",
      "                                  files\n",
      "  --autoscaling-config TEXT       the file that contains the autoscaling\n",
      "                                  config\n",
      "  --no-redirect-output            do not redirect non-worker stdout and stderr\n",
      "                                  to files\n",
      "  --plasma-store-socket-name TEXT\n",
      "                                  manually specify the socket name of the\n",
      "                                  plasma store\n",
      "  --raylet-socket-name TEXT       manually specify the socket path of the\n",
      "                                  raylet process\n",
      "  --ray-debugger-external         Make the Ray debugger available externally\n",
      "                                  to the node. This is onlysafe to activate if\n",
      "                                  the node is behind a firewall.\n",
      "  --log-style [auto|record|pretty]\n",
      "                                  If 'pretty', outputs with formatting and\n",
      "                                  color. If 'record', outputs record-style\n",
      "                                  without formatting. 'auto' defaults to\n",
      "                                  'pretty', and disables pretty logging if\n",
      "                                  stdin is *not* a TTY.\n",
      "  --log-color [auto|false|true]   Use color logging. Auto enables color\n",
      "                                  logging if stdout is a TTY.\n",
      "  -v, --verbose\n",
      "  --help                          Show this message and exit.\n",
      "\u001b[0m"
     ]
    }
   ],
   "source": [
    "!ray start --help"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## ray --version\n",
    "\n",
    "Show the version of Ray you are using."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "ray, version 1.11.0\n",
      "\u001b[0m"
     ]
    }
   ],
   "source": [
    "!ray --version"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "If Ray is running on this node, the output can be very long. It shows the status of the nodes, running worker processes and various other Python processes being executed, and [Redis](https://redis.io/) processes, which are used as part of the distributed object store for Ray. We discuss these services in greater detail in the [Advance Ray tutorial](../advanced-ray/00-Advanced-Ray-Overview.ipynb)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "If there are multiple Ray instances running on this node, you'll have to specify the correct address. Run `ray stat` to see a list of those addresses, then pick the correct one:\n",
    "\n",
    "```shell\n",
    "ray stat --address IP:PORT\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "`ray stat` returns the exit code `0` if Ray is running locally or a nonzero value if it isn't. The following command exploits this feature and starts a _head_ node for Ray:\n",
    "\n",
    "```shell\n",
    "ray stat > /dev/null 2>&1 || ray start --head\n",
    "```\n",
    "\n",
    "All output of `ray stat` is sent to `/dev/null` (which throws it away) and if the status code is nonzero, then the command after the `||` is executed, `ray start --head`."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You can also get cluster information inside your application using API methods.\n",
    "\n",
    "See [Inspect the Cluster State](\n",
    "https://docs.ray.io/en/latest/package-ref.html#inspect-the-cluster-state) for details."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## ray start and ray stop\n",
    "\n",
    "As shown in the previous cell, `ray start` is used to start the Ray processes on a node. When the `--head` flag is used, it means this is the master node that will be used to bootstrap the cluster. \n",
    "\n",
    "When you want to stop Ray running on a particular node, use `ray stop`.\n",
    "\n",
    "> **WARNING:** Running `ray stop` will impact any Ray applications currently running on this node, including all other lesson notebooks currently running Ray, so if you intend to stop Ray, first save your work, close those notebooks, and stop their processes using the _Running_ tab on the left of the Jupyter Lab UI. The tab might be labelled with a white square surrounded by a dark circle instead of _Running_.  \n",
    "\n",
    "We won't actually run `ray start` or `ray stop` in what follows, to avoid causing problems for other lessons. We'll just describe what they do and the output they print."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "When you run `ray start --head` you see output like the following (unless an error occurs):\n",
    "\n",
    "```shell\n",
    "$ ray start --head\n",
    "2020-05-23 07:47:47,469\tINFO scripts.py:357 -- Using IP address 192.168.1.149 for this node.\n",
    "2020-05-23 07:47:47,489\tINFO resource_spec.py:212 -- Starting Ray with 4.3 GiB memory available for workers and up to 2.17 GiB for objects. You can adjust these settings with ray.init(memory=<bytes>, object_store_memory=<bytes>).\n",
    "2020-05-23 07:47:47,865\tINFO services.py:1170 -- View the Ray dashboard at localhost:8265\n",
    "2020-05-23 07:47:47,912\tINFO scripts.py:387 -- \n",
    "Started Ray on this node. You can add additional nodes to the cluster by calling\n",
    "\n",
    "    ray start --address='192.168.1.149:10552' --redis-password='5241590000000000'\n",
    "\n",
    "from the node you wish to add. You can connect a driver to the cluster from Python by running\n",
    "\n",
    "    import ray\n",
    "    ray.init(address='auto', redis_password='5241590000000000')\n",
    "\n",
    "If you have trouble connecting from a different machine, check that your firewall is configured properly. If you wish to terminate the processes that have been started, run\n",
    "\n",
    "    ray stop\n",
    "```\n",
    "\n",
    "(You'll see a different IP address.)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The output includes a line like this:\n",
    "\n",
    "```shell\n",
    "ray start --address='192.168.1.149:10552' --redis-password='5241590000000000'\n",
    "```\n",
    "\n",
    "This is the `ray start` command you would use on the other machines where you want to start Ray and have them join the same cluster.\n",
    "\n",
    "Note also the instructions for code to add to your application.\n",
    "\n",
    "```python\n",
    "import ray\n",
    "ray.init(address='auto', ignore_reinit_errors=True, redis_password='5241590000000000')\n",
    "```\n",
    "\n",
    "The `redis_password` shown is the default value. We didn't specify this argument when we called `ray.init()` in other notebooks."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You can actually call `ray start --head` multiple times on the same node to create separate clusters. They may appear at first to be a bug, but it is actually useful for testing purposes. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The `ray stop` command usually prints no output. Add the `--verbose` flag for details. \n",
    "\n",
    "> **Warning:** `ray stop` stops all running Ray processes on this node. There is no command line option to specify which one to stop."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## ray memory\n",
    "\n",
    "A new feature of the Ray CLI is the `memory` command which prints a snapshot of the current state of actors and tasks in memory in the cluster. It is useful debugging issues and understanding how Ray has distributed work around your cluster. \n",
    "\n",
    "Here is an example captured on a laptop while the first two lessons in this tutorial were evaluating their cells:\n",
    "\n",
    "```\n",
    "$ ray memory\n",
    "2020-06-26 06:08:55,158\tINFO scripts.py:1042 -- Connecting to Ray instance at 192.168.1.149:6379.\n",
    "WARNING: Logging before InitGoogleLogging() is written to STDERR\n",
    "I0626 06:08:55.163417 90759 489258432 global_state_accessor.cc:25] Redis server address = 192.168.1.149:6379, is test flag = 0\n",
    "I0626 06:08:55.164857 90759 489258432 redis_client.cc:141] RedisClient connected.\n",
    "I0626 06:08:55.167277 90759 489258432 redis_gcs_client.cc:88] RedisGcsClient Connected.\n",
    "I0626 06:08:55.168231 90759 489258432 service_based_gcs_client.cc:75] ServiceBasedGcsClient Connected.\n",
    "-----------------------------------------------------------------------------------------------------\n",
    " Object ID                                Reference Type       Object Size   Reference Creation Site\n",
    "=====================================================================================================\n",
    "; driver pid=89861\n",
    "ffffffffffffffff6ec7e2960c0000c001000000  LOCAL_REFERENCE                ?   (actor call) <ipython-input-7-a62036e0309c>:<module>:7\n",
    "55be66b7df500ad56ec7e2960c0000c003000000  LOCAL_REFERENCE               23   (actor call) <ipython-input-7-a62036e0309c>:<module>:8\n",
    "55be66b7df500ad56ec7e2960c0000c002000000  LOCAL_REFERENCE               15   (actor call) <ipython-input-7-a62036e0309c>:<module>:8\n",
    "ffffffffffffffffffffffff0c00008001000000  LOCAL_REFERENCE               27   (put object) <ipython-input-9-57253d54e26a>:<module>:1\n",
    "0f8aa561996c6719ffffffff0c0000c001000000  LOCAL_REFERENCE               88   (task call) <ipython-input-6-9667649da5b7>:<module>:13\n",
    "55be66b7df500ad56ec7e2960c0000c001000000  LOCAL_REFERENCE               16   (actor call) <ipython-input-7-a62036e0309c>:<module>:8\n",
    "; driver pid=90154\n",
    "aa0e49cf6481351dffffffff100000c001000000  LOCAL_REFERENCE               23   (task call) <ipython-input-17-f5cad4404199>:<module>:1\n",
    "082755fdfe469abcffffffff100000c001000000  LOCAL_REFERENCE                ?   (task call) <ipython-input-31-dd50cc550d0b>:<listcomp>:3\n",
    "57c6dbda70012254ffffffff100000c001000000  LOCAL_REFERENCE                ?   (task call) <ipython-input-31-dd50cc550d0b>:<listcomp>:3\n",
    "fab196f393a5de36ffffffff100000c001000000  LOCAL_REFERENCE               88   (task call) <ipython-input-31-dd50cc550d0b>:<listcomp>:3\n",
    "10473efa8f620095ffffffff100000c001000000  LOCAL_REFERENCE               88   (task call) <ipython-input-31-dd50cc550d0b>:<listcomp>:3\n",
    "dc7dc79e27e8e5b7ffffffff100000c001000000  LOCAL_REFERENCE               23   (task call) <ipython-input-19-e197d2c09385>:<listcomp>:1\n",
    "16053fa58b987ab5ffffffff100000c001000000  LOCAL_REFERENCE                ?   (task call) <ipython-input-31-dd50cc550d0b>:<listcomp>:3\n",
    "852d61559823797effffffff100000c001000000  LOCAL_REFERENCE               23   (task call) <ipython-input-19-e197d2c09385>:<listcomp>:1\n",
    "2e1f2a844f6b2fd4ffffffff100000c001000000  LOCAL_REFERENCE               23   (task call) <ipython-input-19-e197d2c09385>:<listcomp>:1\n",
    "a52080f6c7937c01ffffffff100000c001000000  LOCAL_REFERENCE                ?   (task call) <ipython-input-31-dd50cc550d0b>:<listcomp>:3\n",
    "a1e6529f26e2773cffffffff100000c001000000  LOCAL_REFERENCE                ?   (task call) <ipython-input-31-dd50cc550d0b>:<listcomp>:3\n",
    "9991ac8b6172b3f2ffffffff100000c001000000  LOCAL_REFERENCE               23   (task call) <ipython-input-18-a0b7fb747444>:<module>:1\n",
    "3cdffb6f345ef8f3ffffffff100000c001000000  LOCAL_REFERENCE               88   (task call) <ipython-input-31-dd50cc550d0b>:<listcomp>:3\n",
    "0a51ce9438517c13ffffffff100000c001000000  LOCAL_REFERENCE                ?   (task call) <ipython-input-31-dd50cc550d0b>:<listcomp>:3\n",
    "-----------------------------------------------------------------------------------------------------\n",
    "```\n",
    "\n",
    "All references are local because this is the output for a single machine. There are tasks and actors running in the workers, all of which are associated with _driver_ processes that originate with `ipython` processes used by the notebooks."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## ray status\n",
    "\n",
    "A new feature of the Ray CLI is the `status` command for printing various status information about the cluster.\n"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.12"
  },
  "toc": {
   "base_numbering": 1,
   "nav_menu": {},
   "number_sections": false,
   "sideBar": true,
   "skip_h1_title": false,
   "title_cell": "Table of Contents",
   "title_sidebar": "Contents",
   "toc_cell": false,
   "toc_position": {
    "height": "calc(100% - 180px)",
    "left": "10px",
    "top": "150px",
    "width": "382.391px"
   },
   "toc_section_display": true,
   "toc_window_display": true
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}