Construct Evil Example

[PaulWAyers]

Make an evil example.

The idea would be to find an important function (from the literature) and then sample it very nonuniformly.

We could try the model in the (retracted, but still a model) paper:
https://www.nature.com/articles/s41586-021-04096-9

[PaulWAyers]

The idea is to find an explicit function in the literature,

$$ f(x_1,x_2,\ldots,x_n) $$

which we will then sample. However, we won't sample this uniformly, but rather nonuniformly. (E.g., random numbers but not uniformly distributed.) We'll then see how well diverse-selector works versus random selection.

[PaulWAyers]

Another possibility is cancer risk exposure.
https://www.thelancet.com/cms/10.1016/S0140-6736(22)01438-6/attachment/e14dc624-4fe9-4ce5-8736-bb17be93d0f3/mmc1.pdf
https://www.thelancet.com/journals/lancet/article/PIIS0140-6736(22)01438-6/fulltext#sec1

[PaulWAyers]

We could also use actuarial tables.

There are some online cancer-risk calculators.
https://knowyourchances.cancer.gov/big_picture_charts.html
https://www.calculators.org/health/cancer.php
https://www.sciencedirect.com/science/article/pii/S0090429505010071
https://www.mskcc.org/nomograms
https://riskcalc.org/

[PaulWAyers]

Some GitHub repositories have calculators.
https://github.com/raghav103/Lung_Cancer_Predictor
https://github.com/PanduDcau/Lung-Cancer-Project
https://github.com/ToshY/pca-riskcalculator
https://github.com/advikmaniar/ML-Healthcare-Web-App/tree/main
https://github.com/videntity/python-framingham10yr/tree/master
https://github.com/Jean-njoroge/Breast-cancer-risk-prediction

@xychem will look at these and decide which is best for generating data. The main goal is that it is favorable to

• have lots of descriptors/dimensions.
• have an easy model that we can run to generate lots of data (beyond what is actually in the training data).
• more training data is good.
• more stars/forks on the repository is good. This means one of the last three is best, probably.

I'd prefer the last one if it meets all the other criteria. However, it may be better to use the next-to-last one as it is a simple function that we can use. (It's the easiest choice.) The easiest way to sample nonuniformly would be to pick a variable (say, age) and sample it very inhomogeneously. Then our predictions should be bad unless we sample with diversity.

[PaulWAyers]

It seems that all the calculators on riskcalc.org from the Cleveland Clinic can be accessed in a free-and-open-source way.
https://github.com/orgs/ClevelandClinicQHS/repositories?type=all

There is a lot of data also on risk factors vs incidence at
https://github.com/kritikaparmar-programmer/HealthCheck

[PaulWAyers]

Consider the 4-well potential in the attached papers.

• Sample using random numbers in the range $-3 \le R_1 \le 3$ and $-3 \le R_2 \le 3$. Select a huge number of points (~1e6).
• Sample using Boltzmann sampling. For a given (inverse) temperature, keep each point with probability $\exp[-\beta (E - E_0)]$ where $E_0$ is the lowest energy structure, which has energy 1.780 (see table 1 in the paper. Since the highest reaction barrier is about 1.5 units above the lowest energy, it's reasonable to consider $\beta = 1.5^{-k}$ where k = 0 is a random sample (so it's not necessary), and k=1, k=2, k=3, etc. are increasingly selective samples.

For a given number of samples, $S$, do the following three things:

1. Select $S$ points from the randomly generated data. ($\beta = 0$)
2. Select $S$ points from Boltzmann sampling with different $\beta$. These $S$ points are chosen at random, because the Boltzmann sample already screened based on energy.
3. Select the $S$ points with lowest energy. ($\beta \rightarrow \infty$)

Try to fit the potential energy curve using a method like Gaussian processes. Measure the error based on a regular grid of points in the range $-3 \le R_1 \le 3$ and $-3 \le R_2 \le 3$. I'd generate 100 points in each direction (so 10,000 total points) and keep track of

1. Mean absolute error.
2. Root-mean-square error.
3. Maximum absolute error.

The hypothesis is that the errors get worse when biased sampling is performed, but also that diverse selection helps "cure" some of the problems due to biased sampling. So if we randomly sample $S$ points from the Boltzmann sample, it should be worse than if we diversely selected $S$ points from the Boltzmann sample. This will be especially helpful for reducing outliers.

• Dey, B. K., & Ayers, P. W. (2006). A Hamilton–Jacobi type equation for computing minimum potential energy paths. Molecular Physics, 104(4), 541-558.
• Liu, Y., Burger, S. K., Dey, B. K., Sarkar, U., Janicki, M. R., & Ayers, P. W. (2010). The Fast Marching Method for Determining Chemical Reaction Mechanisms in Complex Systems. Quantum Biochemistry, 171-195.

[PaulWAyers]

For breast cancer dataset.

1. Can you fit an SVM on a (random) subset of the data? How does the error increase ans the quantity of data decreases?Assume, for now, random sampling.
2. Can you get away with less data if you use diverse sampling?
3. If you sub-sample the data with a bias (e.g., reject data where the tumor has a large perimeter or large area, using Boltmann-like screening of the data), can you still fit an SVM? Does the fit work better with a "diverse" sample or a "random" sample?

[xychem]

@PaulWAyers @FanwangM @FarnazH

There are some crude restults on my computers. It seems that:

1. Random sampling is better when k is smaller whatever the maximum error, mean squared error or mean absolute error.
2. The mean squared error and mean abolute error is decay slowly but better than maximum error, and the maximum error looks like not convergent when k is lager and the largest sampling number is 200.
3. Optisim method has smaller maximum error than random sampling and all the maximum error of Optisim method is convergent.

---

Question:

1. Boltzmann sampling : I randomly choose the 1e+6 points and calculate their probabilities, then according their probabilities to choose 1e+3 points and iterate the processure to get 1e+6 points. This is one of reasons that my caculation is so large. I don't know how to do the boltzamann sampling in a more economical method, so I need your help.
2. An account for Comupter Canada: I need an account of Computer Canada.The job is large when sampling number over 100 and it runs slowly on my computer, so I need to submit the jobs to servers. Fanwang tought me how to submit jobs and I also have booked the SHARCNET New User / Refresher Webinar Confirmation of Computer Canada tomorrow.

---

The maximum error with different k by random sampling:

The mean absolute error with different k by random sampling:

The mean squared error with different k by random sampling:

---

The compare with k=0, 10, 20 between opisim sampling and random sampling:

[PaulWAyers]

I'd look at a smaller step in k. k=20 is very aggressive screening.

To sample with Boltzmann probability, you just compute the probability, which is $e^{-\beta(E-E_0)}$, where $E_0 = 1.780$ according to the paper. This gives every point a probability, $0 \le p \le 1$. Then for each point you generate a random number, $r$, between zero and one, and accept the result only if $p \ge r$.

The plots you have are very jumpy, because you need to find a way to make multiple samples of the same size. That's easy for random sampling but hard for most of the methods in DiverseSelector, because (by default) they start with the medoid.

1. The basic idea is to first make a huge sample. I recommended 1e6 points, but you may need more. 1e7 will let you consider a larger value of k, for example.
2. Make a Boltzmann sample for a given $k \cdot \beta$. $k = 0$ corresponds to a random sample. Also consider the 1000 points with lowest energy; this corresponds to $k \rightarrow \infty$. At the end of step 2, you have a sample for each value of k. When the size of the Boltzmann sample is less than 1e3, the sample is too small and cannot be used.
3. Construct sub-samples. For each value of $k$, choose a random sub-sample of 1e3 points. Do this repeatedly (perhaps 10 times), so that you have 10 sub-samples for each $k$. By averaging your results over these sub-samples, you'll help smooth over the "bumps" in the curves.
4. Use random selection, OptiSim, and other algorithms to select $S$ points.
5. Fit using Gaussian Process Regression (or similar).
6. Compute mean-absolute error, root-mean-square error, and maximum absolute error for the regular grid of points from $-3.0 \le x,y \le 3.0$.
7. Average these errors over all of the subsamples from step 3 and plot the result.

I hope this will give smoother plots of error vs. $S$, and show more decisive preferences for various sampling methods. My hypothesis would be that for $k=0$ (random sampling), random selection works well. But as $k$ gets larger, I expect it to be more important to choose diverse selection methods in step 4.

[xychem]

I use the boltzmann sampling method that you say but it needs much time to get 1e7 points (more than several hours when k is large, so I kill the job). Maybe I need an account of CC (Computer Canada).

And there is a question about the symbol of k. If $p=exp(-1.5^{-k} \cdot (E-E_0))$, it means when $k\to\infty$, and $p \to$ 1 which represents random sampling. The previous code I write is $\beta = 1.5^k$, so the database will be biased with the increasement of k. Howerver when k is zero ,the sample is also boltzmann sampling. The previous result is right because I use the random sampling when k=0.

---

New code of boltzmann sampling:

# define the boltzmann sampling method
def boltzmann_sample(k,sample_number):
    '''
        k : int
            the order of 1.5

        sample_number : int
            the number of sample number
    '''

    E_0 = 1.780     # the minimum energy
    sample_list=[]
    
    while len(sample_list) < sample_number: 
        rng = np.random.default_rng()
        rn = rng.random()       # generate a random number
        pd = (rng.random((1,2))-0.5)*6      # generate a point with random method from -3 to 3
        p = np.exp(-((1.5)**(k))*((potential_energy(pd[0,0],pd[0,1]))-E_0))     # calculate the probability of the point
        if  p >= rn:        # if p >= random_number than choose the point
            sample_list.append(pd)
    
    sample_list = np.array(sample_list)[:,0]        # change list to np.array
    return sample_list

[PaulWAyers]

The are instructions on how to make an account on Compute Canada in the BootCamp repository that I think I shared with you when you started. It's good to use Compute Canada. Ideally, you can generate data for a day or two; more data is a good thing.

$k=0$ is random sampling, corresponding to infinite temperature (zero $\beta = 0$).
As $k$ gets larger, we want the temperature to get lower ($\beta$ to get larger). So we want

$$ p = e^{-\beta (E- E_0)} $$

and $\beta = 1.5^k$ is good. Sorry for making a typo there. Do be careful that there is a negative sign in the exponential, so that the probability factor is always between zero and one.

[xychem]

1. So I should use the $p=e^{-k\cdot\beta(E-E0)}$ rather than $p=e^{-\beta(E-E0)}$? The former one will be random sample when k=0.
2. It seems that I need your CCRI to register an account of Compute Canada.
   

[PaulWAyers]

The CCRI is in the bootcamp.