reference/Troubleshooting-Tips-Tricks.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Troubleshooting, Tips, and Tricks\n",
    "\n",
    "© 2019-2020, Anyscale. All Rights Reserved\n",
    "\n",
    "![Anyscale Academy](../images/AnyscaleAcademy_Logo_clearbanner_141x100.png)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Join Us at Ray Summit 2020!\n",
    "\n",
    "Join us for the [_free_ Ray Summit 2020 virtual conference](https://events.linuxfoundation.org/ray-summit/?utm_source=dean&utm_medium=embed&utm_campaign=ray_summit&utm_content=anyscale_academy), September 30 - October 1, 2020. We have an amazing lineup of luminar keynote speakers and breakout sessions on the Ray ecosystem, third-party Ray libraries, and applications of Ray in the real world.\n",
    "\n",
    "<a href=\"https://events.linuxfoundation.org/ray-summit/?utm_source=dean&utm_medium=embed&utm_campaign=ray_summit&utm_content=anyscale_academy\">\n",
    "<img src=\"../images/raysummit-horizontal-white-banner-full.png\" alt=\"Ray Summit 2020\"/>\n",
    "</a>\n",
    "\n",
    "For information about other online events, see [anyscale.com/events](https://anyscale.com/events)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Getting Help\n",
    "\n",
    "* The [#tutorial channel](https://ray-distributed.slack.com/archives/C011ML23W5B) on the [Ray Slack](https://ray-distributed.slack.com). [Click here](https://forms.gle/9TSdDYUgxYs8SA9e8) to join.\n",
    "* [Email](mailto:academy@anyscale.com)\n",
    "\n",
    "Find an issue? Please report it!\n",
    "\n",
    "* [GitHub issues](https://github.com/anyscale/academy/issues)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## For More Information on Ray and Anyscale\n",
    "\n",
    "* [ray.io](https://ray.io): The Ray website. In particular:\n",
    "    * [Documentation](https://ray.readthedocs.io/en/latest/): The full Ray documentation\n",
    "    * [Blog](https://medium.com/distributed-computing-with-ray): The Ray blog\n",
    "    * [GitHub](https://github.com/ray-project/ray): The source code for Ray\n",
    "* [anyscale.com](https://anyscale.com/): The company developing Ray and these tutorials. In particular:\n",
    "    * [Blog](https://anyscale.com/blog/): The Anyscale blog\n",
    "    * [Events](https://anyscale.com/events/): Online events, [Ray Summit](http://raysummit.org), and meetups   \n",
    "    * [Academy](https://anyscale.com/academy/): Training for Ray and Anyscale products    \n",
    "    * [Jobs](https://jobs.lever.co/anyscale): Yes, we're hiring!\n",
    "* Community:\n",
    "    * [Ray Slack](ray-distributed.slack.com) ([Click here](https://forms.gle/9TSdDYUgxYs8SA9e8) to join): The best forum for help on Ray. Use the `#tutorials` channel to ask for help on these tutorials!\n",
    "    * [ray-dev mailing list](https://groups.google.com/forum/?nomobile=true#!forum/ray-dev)\n",
    "    * [@raydistributed](https://twitter.com/raydistributed)\n",
    "    * [@anyscalecompute](https://twitter.com/anyscalecompute)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Troubleshooting\n",
    "\n",
    "### ray.init() Fails\n",
    "\n",
    "Suppose you get an error like this:\n",
    "\n",
    "```\n",
    "... INFO services.py:... -- Failed to connect to the redis server, retrying.\n",
    "```\n",
    "\n",
    "It probably means you are running a VPN on your machine. [At this time](https://github.com/ray-project/ray/issues/6573), you can't use `ray.init()` with a VPN running. You'll have to stop your VPN to run `ray.init()`, then once it finishes, you can restart your VPN.\n",
    "\n",
    "If `ray.init()` still fails, it may be that old Redis processes are somehow still running. On MacOS and Linux systems, use a terminal window and run the following command, shown with example output:\n",
    "\n",
    "```shell\n",
    "$ ps -ef | grep redis\n",
    "501 36029     1   0  1:53PM ??         0:00.03 .../lib/python3.7/site-packages/ray/core/src/ray/thirdparty/redis/src/redis-server *:48044\n",
    "501 36030     1   0  1:53PM ??         0:00.02 .../lib/python3.7/site-packages/ray/core/src/ray/thirdparty/redis/src/redis-server *:42902\n",
    "\n",
    "$ kill 36029 36039\n",
    "```\n",
    "\n",
    "Then try again."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### MacOS Annoyances\n",
    "\n",
    "If `ray.init()` worked (for example, you see a message like _View the Ray dashboard at localhost:8265_) and you're using a Mac, you may get several annoying dialogs asking you if you want to allow incoming connections for `python` and/or `redis-server`. Click \"Accept\" for each one and they shouldn't appear again during this lesson. MacOS is trying to verify if these executables have been properly signed. Ray uses Redis. If you installed Python using Anaconda or other mechanism, then it probably isn't properly signed from the point of view of MacOS. To permanently fix this problem, [see this StackExchange post](https://apple.stackexchange.com/questions/3271/how-to-get-rid-of-firewall-accept-incoming-connections-dialog)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### \"File has changed on disk\"\n",
    "\n",
    "On rare occasions, you'll get a warning that the notebook file you are viewing has change on disk. Usually, you'll just want to keep the file you're viewing in Jupyter, especially if you have already made changes, but reloading it is fine, too."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### \"AssertionError\" While Graphs Are Being Drawn \n",
    "\n",
    "We've observed a few occasions when errors like the following are written in the notebook outputs (very long numbers truncated):\n",
    "\n",
    "```\n",
    "WARNING:param.dynamic_operation: Callable raised \"AssertionError('New option id 43948... does not match any option trees in Store.custom_options.',)\".\n",
    "\n",
    "AssertionError: New option id 11997... does not match any option trees in Store.custom_options.\n",
    "```\n",
    "\n",
    "They are related to the graphing libraries we use and appear to be harmless."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### \"ImportError: No module named gym\"\n",
    "\n",
    "If you are working on a laptop and using Anaconda, and you get import errors for modules like `gym`, e.g., _ImportError: No module named gym_, this probably indicates you have the wrong `python` and `pip` commands on your path ahead of the versions for your Conda environment. To verify, use the `which` command:\n",
    "\n",
    "```shell\n",
    "$ which -a python\n",
    "/usr/bin/python\n",
    "/Users/me/anaconda3/envs/anyscale-academy/bin/python\n",
    "```\n",
    "\n",
    "In this example, the built-in `python` is ahead of the correct Conda environment `python`.\n",
    "\n",
    "To fix, try the following:\n",
    "\n",
    "First, try activiting the `anyscale-academy` environment again: \n",
    "\n",
    "```shell\n",
    "conda activate anyscale-academy\n",
    "```\n",
    "\n",
    "(see the [README](../README.md) for more details about the Conda environment we are using).\n",
    "\n",
    "Run `which -a python` again. Does it show your `~/anaconda3/envs/anyscale-academy/bin/python` first? If not, force the correct ordering by changing the path: \n",
    "\n",
    "```shell\n",
    "export PATH=$HOME/anaconda3/envs/anyscale-academy/bin:$PATH\n",
    "```\n",
    "\n",
    "Run the `which` command again to confirm the correct `python` is now first in the list."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Saving Resources by Ending Sessions You No Longer Need\n",
    "\n",
    "When doing lessons on a machine or cloud instance with low resources, close notebooks you no longer need and stop their kernels. Do the latter by clicking the left-hand icon with a white square inside a black circle. The tool tip says _Running Terminals and Kernels_. Click _SHUT DOWN_ for any notebooks you no longer need."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Tips and Tricks\n",
    "\n",
    "### Jupyter Tips\n",
    "\n",
    "* **Learning Jupyter Lab:** Open _Help > JupyerLab Reference_ for a user guide and detailed information about Jupyter Lab. The _Help_ menu has other references, too.\n",
    "* **Excessive Output:** If the output of a cell is very long, right click on it and select _Enable Scrolling for Outputs_."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### General Task and Actor Tips\n",
    "\n",
    "* To create a task from a function or an actor from a class, annotate it with `@ray.remote`.\n",
    "* Invoke tasks with `foo.remote(...)`\n",
    "* Invoke actor constructors `MyClass.remote(...)`.\n",
    "* Invoke actor methods with `my_instance.remote(...)`.\n",
    "* Invocations return an `ObjectRef` for a _future_. Use `ray.get(id)` to return the value.\n",
    "* However, `ray.get()` blocks, so consider using `ray.wait()` when waiting for a collection of futures, so as they become available, you can retrieve them with `ray.get()` (which won't block when we know the results are available), then process the results, all while waiting for the rest to finish.\n",
    "* Pick functions to parallelize that do enough work so that the Ray \"remoting\" overhead is not significant. Very short functions will actually yield lower performance if convert to tasks. \n",
    "* Similarly, avoid too many actors, as each one is pinned to memory until no longer needed."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Using Existing Functions and Classes\n",
    "\n",
    "An existing function can be used as a task by defining a new task function that calls the original function. For example:\n",
    "\n",
    "```python\n",
    "def work(args):\n",
    "    do_work(...)\n",
    "    \n",
    "@ray.remote\n",
    "def remote_work(args):\n",
    "    do_work(args)\n",
    "```\n",
    "\n",
    "This allows you to use either version, as appropriate."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Similarly, existing classes can be subclassed to create actors:\n",
    "\n",
    "```python\n",
    "class Counter():\n",
    "    def __init__(self, init_count):\n",
    "        self.count = init_count\n",
    "        \n",
    "    def increment():\n",
    "        self.count +=1\n",
    "        return self.count\n",
    "    \n",
    "@ray.remote\n",
    "class RemoteCounter(Counter):\n",
    "    def __init__(self, init_count):\n",
    "        super().__init__(init_count)\n",
    "        \n",
    "    def get_count():\n",
    "        return self.count\n",
    "```\n",
    "\n",
    "Note that we added a `get_count()` method, because member attributes can't be accessed directly, in contrast with normal classes. Of course, you don't need to add _accessor_ methods unless you need them."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Using the Ray Dashboard\n",
    "\n",
    "#### Opening the Dashboard\n",
    "\n",
    "As it executes, `ray.init` prints the dashboard URL.\n",
    "\n",
    "You can get the URL later if needed using `ray.get_webui_url()`.\n",
    "\n",
    "> **Tip:** In a notebook, use a cell with `f'http://{ray.get_webui_url()}'`. The full URL printed will be clickable."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Profiling Actors\n",
    "\n",
    "The _Logical View_ offers a powerful and convenient way to profile actor performance using [flame graphs](http://www.brendangregg.com/flamegraphs.html). Details are in the [Dashboard docs](https://ray.readthedocs.io/en/latest/ray-dashboard.html#ray-dashboard).\n",
    "\n",
    "This feature uses [py-spy](https://github.com/benfred/py-spy) to instrument and profile the application. Unfortunately, you may be asked to enter the `sudo` password to use this feature, because of the way it instruments processes. Currently, the only way to get this to work with the Dashboard launched from notebooks is to use _passwordless sudo_. On MacOS and Linux systems, it should be sufficient to add a line like the following the `/etc/sudoers` (edited using `sudo visudo`!):\n",
    "\n",
    "```\n",
    "yourusername ALL = (ALL) NOPASSWD: ALL\n",
    "```\n",
    "\n",
    "Carefully the consider the security implications of this change!!\n",
    "\n",
    "The alternative is to run your Python application from a command line, then open the Dashboard. When you click a link to profile code, as discussed next, you'll be prompted for your password in the terminal. (The prompt could get mixed with other output from the program.) Enter your password there and the profiling will continue.\n",
    "\n",
    "Using either approach, to profile with the Dashboard, click the _Logical View_ tab. It shows a list of actors that have been run or are running. Find the running actor that appears to be the one you want to profile. You'll see a line like this:\n",
    "\n",
    "> Actor <hex_number> (Profile for 10s 30s 60s) Kill Actor\n",
    "\n",
    "The _10s, 30s, 60s_ are links. Click the time you want (pick one that's shorter than the remaining execution time). \n",
    "\n",
    "When it finishes, click _Profile results_. A new tab opens with the _speedscope_ view of the data, which shows a flame graph. You can learn more about navigating and using this tool at the [speedscope GitHub site](https://github.com/jlfwong/speedscope)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "A lot of the data will be related to actor messaging and not normally interesting. Look at the following screen shot, cropped from a screen shot in [lesson 4]():\n",
    "\n",
    "![Conway's GoL Flame Graph](../images/ConwaysGameOfLife-FlameGraph-crop.png)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Note the arrow and pagination on the upper-right hand side. Sometimes navigating pages will take you to interesting data. Also, the _Left Heavy_ button on the upper-left hand side is clicked, so the view is zoomed into the interesting data about the `step()` method profiled here."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Profiling Code with ray.timeline()\n",
    "\n",
    "The other built-in way to profile performance uses `ray.timeline(file)` ([documentation](https://ray.readthedocs.io/en/latest/package-ref.html#ray.timeline)). It requires a Chrome web browser to view the data. This is the only way to profile tasks.\n",
    "\n",
    "Use it as follows:\n",
    "\n",
    "```\n",
    "ray.timeline('timeline.txt')\n",
    "my_long_task.remote(...)  # task to profile\n",
    "```\n",
    "\n",
    "Then, open chrome://tracing in the Chrome web browser (only Chrome is supported) and click the _load_ button to load the file. To zoom in or out, click the asymmetric up-down arrow button. To move around, click the crossed arrow and drag a section in view. Click on a box in the timeline to see details about it. \n",
    "\n",
    "Look for blocks corresponding to long-running tasks and look for idle periods, which reflect processing outside the context of Ray."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Cleaning Up\n",
    "\n",
    "When your driver script exists, it will implicitly call `ray.shutdown()`. This can be useful to do explicitly if you want to restart Ray, for example, in a notebook environment, like this:\n",
    "\n",
    "```python\n",
    "ray.shutdown()\n",
    "ray.init(...)\n",
    "```"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.6"
  },
  "toc": {
   "base_numbering": 1,
   "nav_menu": {},
   "number_sections": false,
   "sideBar": true,
   "skip_h1_title": false,
   "title_cell": "Table of Contents",
   "title_sidebar": "Contents",
   "toc_cell": false,
   "toc_position": {
    "height": "calc(100% - 180px)",
    "left": "10px",
    "top": "150px",
    "width": "382.391px"
   },
   "toc_section_display": true,
   "toc_window_display": true
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}