{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "093c184f-c3d4-4dd8-83b1-517cad4acf50",
   "metadata": {},
   "source": [
    "# Performance Evaluation\n",
    "\n",
    "In this notebook we explore the performance evaluation tools of ```cbxpy```. These are provided in the module ```cbx.utils.success```. For a test problem, we consider the Rastrigin function in dimension ```d=20```. We employ ```N=40``` particles and perform ```M=100``` runs."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "id": "8e52898b-d5d9-40d2-ad82-c816430f926f",
   "metadata": {},
   "outputs": [],
   "source": [
    "%load_ext autoreload\n",
    "%autoreload 2\n",
    "import cbx\n",
    "import cbx.dynamics as dyns\n",
    "import cbx.utils.success as success\n",
    "import numpy as np\n",
    "\n",
    "f = cbx.objectives.Rastrigin()\n",
    "M = 100\n",
    "N = 40\n",
    "d = 20\n",
    "\n",
    "x = np.random.uniform(-3,3,(M,N,d))\n",
    "kwargs = {'x':x, 'sigma':10.1, 'verbosity':0, 'max_it':5000, 'noise':'anisotropic', 'alpha':30, 'dt':0.01, 'f_dim':'3D'}\n",
    "x_true = np.zeros((x.shape[-1],))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f14c54bc-a412-4e37-93f4-4cee8d2fe243",
   "metadata": {},
   "source": [
    "## Perform optimization\n",
    "\n",
    "Using the keywords from above, we define a dynamic and run the optimization."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "id": "1e83d010-2d5f-42e0-9bd8-99ab619df68d",
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "dyn = dyns.CBO(f,**kwargs)\n",
    "dyn.optimize();"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "550d27a5-fa91-40e8-96e7-059b7c9205da",
   "metadata": {},
   "source": [
    "## Performance evaluation\n",
    "\n",
    "To evaluate the performance, we use the ```evaluation``` class, which accepts a list of performance criteria. Each criterion should have a call function that accepts a dynamic as the input and outputs a dictionary containing the following key-value pairs:\n",
    "\n",
    "* ```rate```: the success rate,\n",
    "* ```num```: the absolute number of successful runs,\n",
    "* ```idx```: the indices of the successful runs.\n",
    "\n",
    "We can directly apply this to our dynamic to get a result. Here we use the criterion ```dist_to_min```, which needs the true minimum as the input. This criterion will be satisfied if\n",
    "\n",
    "$$\\|x_{\\text{true}} - x\\|_p < \\texttt{tol},$$\n",
    "\n",
    "where the tolerance and $p$ of the norm can also be specified."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "id": "bfca0359-1c09-4c17-8ac5-f36539b2b542",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "------------------------------\n",
      "Results of success evaluation:\n",
      "Success Rate: 1.0\n",
      "Succesful runs: 100\n",
      "Succesful idx: [ 0  1  2  3  4  5  6  7  8  9 10 11 12 13 14 15 16 17 18 19 20 21 22 23\n",
      " 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47\n",
      " 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71\n",
      " 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95\n",
      " 96 97 98 99]\n"
     ]
    }
   ],
   "source": [
    "seval = success.evaluation(criteria = [success.dist_to_min(x_true, tol=0.25, p=float('inf'))])\n",
    "seval(dyn);"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.12.3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}