This is an experiment in using optimization of random weighted combinations of criteria to identify the Pareto efficient frontier of a multicriterion decision problem. The specifics of the problem are simple. A number of binary decisions are to be made, without constraints, and with each decision evaluated independently on a set of criteria.

In what follows, decisions will be labeled “D1”, “D2”, … and criteria will be labeled “C1”, “C2”, … To facilitate computations, we assume that all criteria are to be maximized. (Criteria that should be minimized will be negated.) We also assume that all criteria are normalized to either the interval [0, 1] or the interval [-1, 0].

Preparation

We first load any required libraries.

library(tidyverse)

Data Generation

Preliminaries

Specify the number of decisions and the number of criteria here. Bear in mind that the solution space will have cardinality 2^(# of decisions), so do not go crazy. Also specify a seed for the random number generator, for reproducibility, and the number of attempts to make with the random solution identifier.

nDecisions <- 10     # number of decisions
nCriteria <- 5       # number of criteria
nTrials <- 1000000   # number of attempts to randomly identify Pareto solutions
criterionNames <- paste0("C", 1:nCriteria)  # name the criteria
seed <- 21419        # random number seed

Seed the random number generator.

set.seed(seed)

After all computations are done, the notebook will plot selected two-dimensional slices of the Pareto frontier. List pairs of criterion indices for the desired plots.

frontierPlots <- list(c(1, 2), c(3, 4), c(2, 5), c(1, 3))  # pairs of criteria to plot

Decisions

We create a tibble of decisions and their criterion values. The criterion values will be randomly generated (uniform over (0, 1)), and some (but not all) will be reversed to give a mix of “more is better” and “less is better” criteria.

# Create the tibble.
decisions <- tibble(ID = paste0("D", 1:nDecisions))
# Set the criterion names.
cNames <- paste0("C", 1:nCriteria)
# Add random variables for the criteria.
for (v in cNames) {
  decisions[, v] <- runif(nDecisions)
}
# Decide how many criteria to invert (at least 1, at most nCriteria - 1).
nInvert <- 1 + rbinom(1, nCriteria - 2, 0.5)
# Select the criteria to invert.
invert <- sample(cNames, nInvert)
# Invert them.
decisions[, invert] <- -decisions[, invert]

Solutions

Now we create a tibble of all possible combinations of decisions. The ID (key) will be an integer whose binary expansion gives the vector of decisions. The “DominatedBy” column will give the ID of a combination that dominates the current combination (or -1 if the current combination is Pareto efficient).

solutions <- tibble(ID = 0:(2^nDecisions - 1), DominatedBy = -1)

We will need a couple of functions to convert between integer ID values and binary vectors.

# Create a vector of powers of 2 (for use in conversions from binary vectors to integers).
powers.of.two <- 2^(0:(nDecisions - 1))
# Convert an ID (integer) to a binary vector of appropriate length. Note that the vector is reversed so that the lowest order bit (corresponding to the first decision) comes first.
fromID <- function(id) { as.integer(head(intToBits(id), nDecisions)) }
# Convert a binary vector of appropriate length to an ID value (integer).
toID <- function(vec) { as.integer(vec %*% powers.of.two) }

The value of each criterion for a solution is the sum of the criterion values for the “yes” (1) decisions in that solution.

# Compute the total value of a criterion (specified by name) for a given solution (specified by ID).
combinedValue <- function(criterion, id) { pull(decisions, criterion) %*% fromID(id) }
# Evaluate every solution on every criterion.
for (v in cNames) {
  solutions[, v] <- sapply(solutions$ID, function(id) combinedValue(v, id) )
}

Computing the Pareto efficient set

Dominance

We first sort the solution table in lexicographically decreasing order according to the criteria. This guarantees that the first solution in the sort order is nondominated. Moreover, if we remove a nondominated solution from the top of the list, and also remove all solutions it dominates, the top solution in the remaining portion of the list is again guaranteed not to be dominated.

The first step is to create a function to lexically sort a set of solutions (specified by their ID values).

# We create a function that takes a list of solution IDs and outputs the IDs in descending lexicographic sort order according to the criteria.
lex.order <- function(ids) {
  x <- solutions %>%
         filter(ID %in% ids) %>%
         select(cNames) %>% 
         do.call(order, .) %>% 
         rev
  ids[x]
}

We also need a function that compares two solutions (specified by IDs) and determines whether the first solution dominates the second one. Note that if the first solution precedes the second solution in our

dominates <- function(first, second) {
  f <- solutions[first + 1, ] %>% select(cNames) %>% as.numeric()
  s <- solutions[second + 1, cNames] %>% as.numeric()
  all(f >= s) && any(f > s)
}

Find the Pareto efficient solutions

Our “to-do” list will be the full set of IDs, in sorted order, along with their domination flags (to save some look-ups).

system.time({
  temp <- lex.order(solutions$ID)
 todo <- solutions %>% select(ID, DominatedBy) %>% arrange(temp)
})
   user  system elapsed 
  0.003   0.000   0.013 

While the to-do list has length greater than one, we peel off the first ID (which corresponds to a nondominated solution), find the IDs of any solutions it dominates, mark them dominated in the solutions tibble and remove them from the to-do list. This will likely be the slowest part of the code!

cat("Searching for Pareto efficient solutions ...")
Searching for Pareto efficient solutions ...
system.time({
  for (i in 1:(nrow(todo) - 1)) {
    id1 <- todo[[i, "ID"]]
    # Process the i-th solution only if it is still not dominated.
    if (todo[[i, "DominatedBy"]] < 0) {
      # Compare to every subsequent solution that is not (yet) dominated.
      for (j in (i + 1):nrow(todo)) {
        if (todo[[j, "DominatedBy"]] < 0) {
          id2 <- todo[[j, "ID"]]
          # If the i-th solution dominates this one, update both the solutions tibble and the to-do list.
          if (dominates(id1, id2)) {
            solutions[[id2 + 1, "DominatedBy"]] <- id1
            todo[[j, "DominatedBy"]] <- id1
          }
        }
      }
    }
  }
})
   user  system elapsed 
312.099   0.277 313.646 

We can now report on the Pareto efficient solution set.

pareto <- solutions %>% filter(DominatedBy == -1) %>% pull(ID)
cat("Number of Pareto efficient solutions = ", length(pareto))
Number of Pareto efficient solutions =  623

Randomly search for Pareto solutions

To randomly find a Pareto efficient solution, we form for each decision a weighted combination of its criterion values (using positive weights uniformly distributed in the interval (0, 1)). Any decision with positive weight is set to 1, and any decision with negative weight is set to zero.

# To avoid repetition, we will extract the criterion values from the decisions tibble to a matrix.
dMatrix <- decisions %>% select(cNames) %>% as.matrix()
# This function generates a random set of weights, forms the combined scores for all decisions, selects the decisions with positive scores, and returns the ID of the corresponding solution.
winner <- function() {
  weights <- runif(nCriteria)
  scores <- as.vector(dMatrix %*% weights)
  chosen <- scores > 0
  toID(chosen)
}

To keep score, we will add a column to the solutions tibble indicating how many times the solution has been identified as Pareto efficient by the random search.

solutions$Identified <- 0

We now run the specified number of trials, reporting the required time.

cat("Running ", nTrials, " trials ...")
Running  1e+06  trials ...
system.time(hits <- replicate(nTrials, winner()))
   user  system elapsed 
  5.798   0.004   5.803 

We tabulate the number of times each solution key was encountered and update the “Identified” column of the solutions data frame.

temp <- as.data.frame(table(hits))
colnames(temp) <- c("Key", "Frequency")
solutions[solutions$ID %in% temp$Key, "Identified"] <- temp$Frequency

Finally, we report how many Pareto efficient solutions were correctly (or incorrectly) detected.

n <- solutions %>% filter(Identified > 0 & DominatedBy == -1) %>% nrow()
cat("Correctly detected ", n, " Pareto efficient solutions")
Correctly detected  126  Pareto efficient solutions
n <- solutions %>% filter(Identified == 0 & DominatedBy == -1) %>% nrow()
cat("\nFailed to detect ", n, " Pareto efficient solutions")

Failed to detect  497  Pareto efficient solutions
n <- solutions %>% filter(Identified > 0 & DominatedBy >= 0) %>% nrow()
cat("\nFalsely identified ", n, " dominated solutions as Pareto efficient")

Falsely identified  0  dominated solutions as Pareto efficient

Frontier plots

Add a factor indicating whether or not a Pareto solution was found by the random search.

solutions <- solutions %>% mutate(Found = ifelse(Identified > 0, "Found", "Not found"))

Define a function to plot any pair of criteria for the Pareto efficient points.

# Given a pair of criterion indices, scatter plot those criteria for the Pareto efficient solutions.
plotPair <- function(pair) {
  # Get the criterion names.
  names <- criterionNames[pair]
  # Drill down to the relevant data.
  temp <- solutions %>% filter(DominatedBy < 0) %>% select(names, Found)
  show(ggplot(data = temp, mapping = aes_string(x = names[1], y = names[2], color = "Found")) + geom_point())
}

Plot the calculated and identified Pareto frontier for each selected pair of criteria. Note that, in all plots, the upper right corner is ideal.

plot.list <- lapply(frontierPlots, plotPair)

Search efficacy

To see how often the random search finds already identified points, we can plot a histogram of the number of times each point was found. We restrict to those that were found at least once.

solutions %>% filter(Found == "Found") %>% ggplot(mapping = aes(x = Identified)) + geom_freqpoly(bins = 50)

Clearly some points are identified quite a few times. Here are the quartiles for the distribution of the number of times each Pareto efficient solution found by the random search was hit.

solutions %>% filter(Found == "Found") %>% select(Identified) %>% summary()
   Identified       
 Min.   :     1.00  
 1st Qu.:    70.25  
 Median :   663.00  
 Mean   :  7936.51  
 3rd Qu.:  4336.75  
 Max.   :203690.00  

The distribution has a very long right tail (significant positive skew).

LS0tCnRpdGxlOiAiRXhwbG9yaW5nIGEgUGFyZXRvIEZyb250aWVyIgpvdXRwdXQ6IGh0bWxfbm90ZWJvb2sKLS0tCgpUaGlzIGlzIGFuIGV4cGVyaW1lbnQgaW4gdXNpbmcgb3B0aW1pemF0aW9uIG9mIHJhbmRvbSB3ZWlnaHRlZCBjb21iaW5hdGlvbnMgb2YgY3JpdGVyaWEgdG8gaWRlbnRpZnkgdGhlIFBhcmV0byBlZmZpY2llbnQgZnJvbnRpZXIgb2YgYSBtdWx0aWNyaXRlcmlvbiBkZWNpc2lvbiBwcm9ibGVtLiBUaGUgc3BlY2lmaWNzIG9mIHRoZSBwcm9ibGVtIGFyZSBzaW1wbGUuIEEgbnVtYmVyIG9mIGJpbmFyeSBkZWNpc2lvbnMgYXJlIHRvIGJlIG1hZGUsIHdpdGhvdXQgY29uc3RyYWludHMsIGFuZCB3aXRoIGVhY2ggZGVjaXNpb24gZXZhbHVhdGVkIGluZGVwZW5kZW50bHkgb24gYSBzZXQgb2YgY3JpdGVyaWEuCgpJbiB3aGF0IGZvbGxvd3MsIGRlY2lzaW9ucyB3aWxsIGJlIGxhYmVsZWQgIkQxIiwgIkQyIiwgLi4uIGFuZCBjcml0ZXJpYSB3aWxsIGJlIGxhYmVsZWQgIkMxIiwgIkMyIiwgLi4uIFRvIGZhY2lsaXRhdGUgY29tcHV0YXRpb25zLCB3ZSBhc3N1bWUgdGhhdCBhbGwgY3JpdGVyaWEgYXJlIHRvIGJlIG1heGltaXplZC4gKENyaXRlcmlhIHRoYXQgc2hvdWxkIGJlIG1pbmltaXplZCB3aWxsIGJlIG5lZ2F0ZWQuKSBXZSBhbHNvIGFzc3VtZSB0aGF0IGFsbCBjcml0ZXJpYSBhcmUgbm9ybWFsaXplZCB0byBlaXRoZXIgdGhlIGludGVydmFsIFswLCAxXSBvciB0aGUgaW50ZXJ2YWwgWy0xLCAwXS4KCiMgUHJlcGFyYXRpb24KCldlIGZpcnN0IGxvYWQgYW55IHJlcXVpcmVkIGxpYnJhcmllcy4KCmBgYHtyfQpsaWJyYXJ5KHRpZHl2ZXJzZSkKYGBgCgojIERhdGEgR2VuZXJhdGlvbgoKIyMgUHJlbGltaW5hcmllcwoKU3BlY2lmeSB0aGUgbnVtYmVyIG9mIGRlY2lzaW9ucyBhbmQgdGhlIG51bWJlciBvZiBjcml0ZXJpYSBoZXJlLiBCZWFyIGluIG1pbmQgdGhhdCB0aGUgc29sdXRpb24gc3BhY2Ugd2lsbCBoYXZlIGNhcmRpbmFsaXR5IDJeKCMgb2YgZGVjaXNpb25zKSwgc28gZG8gbm90IGdvIGNyYXp5LiBBbHNvIHNwZWNpZnkgYSBzZWVkIGZvciB0aGUgcmFuZG9tIG51bWJlciBnZW5lcmF0b3IsIGZvciByZXByb2R1Y2liaWxpdHksIGFuZCB0aGUgbnVtYmVyIG9mIGF0dGVtcHRzIHRvIG1ha2Ugd2l0aCB0aGUgcmFuZG9tIHNvbHV0aW9uIGlkZW50aWZpZXIuCgpgYGB7cn0KbkRlY2lzaW9ucyA8LSAxMCAgICAgIyBudW1iZXIgb2YgZGVjaXNpb25zCm5Dcml0ZXJpYSA8LSA1ICAgICAgICMgbnVtYmVyIG9mIGNyaXRlcmlhCm5UcmlhbHMgPC0gMTAwMDAwMCAgICMgbnVtYmVyIG9mIGF0dGVtcHRzIHRvIHJhbmRvbWx5IGlkZW50aWZ5IFBhcmV0byBzb2x1dGlvbnMKY3JpdGVyaW9uTmFtZXMgPC0gcGFzdGUwKCJDIiwgMTpuQ3JpdGVyaWEpICAjIG5hbWUgdGhlIGNyaXRlcmlhCnNlZWQgPC0gMjE0MTkgICAgICAgICMgcmFuZG9tIG51bWJlciBzZWVkCmBgYAoKU2VlZCB0aGUgcmFuZG9tIG51bWJlciBnZW5lcmF0b3IuCgpgYGB7cn0Kc2V0LnNlZWQoc2VlZCkKYGBgCgpBZnRlciBhbGwgY29tcHV0YXRpb25zIGFyZSBkb25lLCB0aGUgbm90ZWJvb2sgd2lsbCBwbG90IHNlbGVjdGVkIHR3by1kaW1lbnNpb25hbCBzbGljZXMgb2YgdGhlIFBhcmV0byBmcm9udGllci4gTGlzdCBwYWlycyBvZiBjcml0ZXJpb24gaW5kaWNlcyBmb3IgdGhlIGRlc2lyZWQgcGxvdHMuCgpgYGB7cn0KZnJvbnRpZXJQbG90cyA8LSBsaXN0KGMoMSwgMiksIGMoMywgNCksIGMoMiwgNSksIGMoMSwgMykpICAjIHBhaXJzIG9mIGNyaXRlcmlhIHRvIHBsb3QKYGBgCgojIyBEZWNpc2lvbnMKCldlIGNyZWF0ZSBhIHRpYmJsZSBvZiBkZWNpc2lvbnMgYW5kIHRoZWlyIGNyaXRlcmlvbiB2YWx1ZXMuIFRoZSBjcml0ZXJpb24gdmFsdWVzIHdpbGwgYmUgcmFuZG9tbHkgZ2VuZXJhdGVkICh1bmlmb3JtIG92ZXIgKDAsIDEpKSwgYW5kIHNvbWUgKGJ1dCBub3QgYWxsKSB3aWxsIGJlIHJldmVyc2VkIHRvIGdpdmUgYSBtaXggb2YgIm1vcmUgaXMgYmV0dGVyIiBhbmQgImxlc3MgaXMgYmV0dGVyIiBjcml0ZXJpYS4KCmBgYHtyfQojIENyZWF0ZSB0aGUgdGliYmxlLgpkZWNpc2lvbnMgPC0gdGliYmxlKElEID0gcGFzdGUwKCJEIiwgMTpuRGVjaXNpb25zKSkKIyBTZXQgdGhlIGNyaXRlcmlvbiBuYW1lcy4KY05hbWVzIDwtIHBhc3RlMCgiQyIsIDE6bkNyaXRlcmlhKQojIEFkZCByYW5kb20gdmFyaWFibGVzIGZvciB0aGUgY3JpdGVyaWEuCmZvciAodiBpbiBjTmFtZXMpIHsKICBkZWNpc2lvbnNbLCB2XSA8LSBydW5pZihuRGVjaXNpb25zKQp9CiMgRGVjaWRlIGhvdyBtYW55IGNyaXRlcmlhIHRvIGludmVydCAoYXQgbGVhc3QgMSwgYXQgbW9zdCBuQ3JpdGVyaWEgLSAxKS4KbkludmVydCA8LSAxICsgcmJpbm9tKDEsIG5Dcml0ZXJpYSAtIDIsIDAuNSkKIyBTZWxlY3QgdGhlIGNyaXRlcmlhIHRvIGludmVydC4KaW52ZXJ0IDwtIHNhbXBsZShjTmFtZXMsIG5JbnZlcnQpCiMgSW52ZXJ0IHRoZW0uCmRlY2lzaW9uc1ssIGludmVydF0gPC0gLWRlY2lzaW9uc1ssIGludmVydF0KYGBgCgojIyBTb2x1dGlvbnMKCk5vdyB3ZSBjcmVhdGUgYSB0aWJibGUgb2YgYWxsIHBvc3NpYmxlIGNvbWJpbmF0aW9ucyBvZiBkZWNpc2lvbnMuIFRoZSBJRCAoa2V5KSB3aWxsIGJlIGFuIGludGVnZXIgd2hvc2UgYmluYXJ5IGV4cGFuc2lvbiBnaXZlcyB0aGUgdmVjdG9yIG9mIGRlY2lzaW9ucy4gVGhlICJEb21pbmF0ZWRCeSIgY29sdW1uIHdpbGwgZ2l2ZSB0aGUgSUQgb2YgYSBjb21iaW5hdGlvbiB0aGF0IGRvbWluYXRlcyB0aGUgY3VycmVudCBjb21iaW5hdGlvbiAob3IgLTEgaWYgdGhlIGN1cnJlbnQgY29tYmluYXRpb24gaXMgUGFyZXRvIGVmZmljaWVudCkuCgpgYGB7cn0Kc29sdXRpb25zIDwtIHRpYmJsZShJRCA9IDA6KDJebkRlY2lzaW9ucyAtIDEpLCBEb21pbmF0ZWRCeSA9IC0xKQpgYGAKCldlIHdpbGwgbmVlZCBhIGNvdXBsZSBvZiBmdW5jdGlvbnMgdG8gY29udmVydCBiZXR3ZWVuIGludGVnZXIgSUQgdmFsdWVzIGFuZCBiaW5hcnkgdmVjdG9ycy4KCmBgYHtyfQojIENyZWF0ZSBhIHZlY3RvciBvZiBwb3dlcnMgb2YgMiAoZm9yIHVzZSBpbiBjb252ZXJzaW9ucyBmcm9tIGJpbmFyeSB2ZWN0b3JzIHRvIGludGVnZXJzKS4KcG93ZXJzLm9mLnR3byA8LSAyXigwOihuRGVjaXNpb25zIC0gMSkpCiMgQ29udmVydCBhbiBJRCAoaW50ZWdlcikgdG8gYSBiaW5hcnkgdmVjdG9yIG9mIGFwcHJvcHJpYXRlIGxlbmd0aC4gTm90ZSB0aGF0IHRoZSB2ZWN0b3IgaXMgcmV2ZXJzZWQgc28gdGhhdCB0aGUgbG93ZXN0IG9yZGVyIGJpdCAoY29ycmVzcG9uZGluZyB0byB0aGUgZmlyc3QgZGVjaXNpb24pIGNvbWVzIGZpcnN0Lgpmcm9tSUQgPC0gZnVuY3Rpb24oaWQpIHsgYXMuaW50ZWdlcihoZWFkKGludFRvQml0cyhpZCksIG5EZWNpc2lvbnMpKSB9CiMgQ29udmVydCBhIGJpbmFyeSB2ZWN0b3Igb2YgYXBwcm9wcmlhdGUgbGVuZ3RoIHRvIGFuIElEIHZhbHVlIChpbnRlZ2VyKS4KdG9JRCA8LSBmdW5jdGlvbih2ZWMpIHsgYXMuaW50ZWdlcih2ZWMgJSolIHBvd2Vycy5vZi50d28pIH0KYGBgCgpUaGUgdmFsdWUgb2YgZWFjaCBjcml0ZXJpb24gZm9yIGEgc29sdXRpb24gaXMgdGhlIHN1bSBvZiB0aGUgY3JpdGVyaW9uIHZhbHVlcyBmb3IgdGhlICJ5ZXMiICgxKSBkZWNpc2lvbnMgaW4gdGhhdCBzb2x1dGlvbi4KCmBgYHtyfQojIENvbXB1dGUgdGhlIHRvdGFsIHZhbHVlIG9mIGEgY3JpdGVyaW9uIChzcGVjaWZpZWQgYnkgbmFtZSkgZm9yIGEgZ2l2ZW4gc29sdXRpb24gKHNwZWNpZmllZCBieSBJRCkuCmNvbWJpbmVkVmFsdWUgPC0gZnVuY3Rpb24oY3JpdGVyaW9uLCBpZCkgeyBwdWxsKGRlY2lzaW9ucywgY3JpdGVyaW9uKSAlKiUgZnJvbUlEKGlkKSB9CiMgRXZhbHVhdGUgZXZlcnkgc29sdXRpb24gb24gZXZlcnkgY3JpdGVyaW9uLgpmb3IgKHYgaW4gY05hbWVzKSB7CiAgc29sdXRpb25zWywgdl0gPC0gc2FwcGx5KHNvbHV0aW9ucyRJRCwgZnVuY3Rpb24oaWQpIGNvbWJpbmVkVmFsdWUodiwgaWQpICkKfQpgYGAKCiMgQ29tcHV0aW5nIHRoZSBQYXJldG8gZWZmaWNpZW50IHNldAoKIyMgRG9taW5hbmNlCgpXZSBmaXJzdCBzb3J0IHRoZSBzb2x1dGlvbiB0YWJsZSBpbiBsZXhpY29ncmFwaGljYWxseSBkZWNyZWFzaW5nIG9yZGVyIGFjY29yZGluZyB0byB0aGUgY3JpdGVyaWEuIFRoaXMgZ3VhcmFudGVlcyB0aGF0IHRoZSBmaXJzdCBzb2x1dGlvbiBpbiB0aGUgc29ydCBvcmRlciBpcyBub25kb21pbmF0ZWQuIE1vcmVvdmVyLCBpZiB3ZSByZW1vdmUgYSBub25kb21pbmF0ZWQgc29sdXRpb24gZnJvbSB0aGUgdG9wIG9mIHRoZSBsaXN0LCAqYW5kIGFsc28gcmVtb3ZlIGFsbCBzb2x1dGlvbnMgaXQgZG9taW5hdGVzKiwgdGhlIHRvcCBzb2x1dGlvbiBpbiB0aGUgcmVtYWluaW5nIHBvcnRpb24gb2YgdGhlIGxpc3QgaXMgYWdhaW4gZ3VhcmFudGVlZCBub3QgdG8gYmUgZG9taW5hdGVkLgoKVGhlIGZpcnN0IHN0ZXAgaXMgdG8gY3JlYXRlIGEgZnVuY3Rpb24gdG8gbGV4aWNhbGx5IHNvcnQgYSBzZXQgb2Ygc29sdXRpb25zIChzcGVjaWZpZWQgYnkgdGhlaXIgSUQgdmFsdWVzKS4KCmBgYHtyfQojIFdlIGNyZWF0ZSBhIGZ1bmN0aW9uIHRoYXQgdGFrZXMgYSBsaXN0IG9mIHNvbHV0aW9uIElEcyBhbmQgb3V0cHV0cyB0aGUgSURzIGluIGRlc2NlbmRpbmcgbGV4aWNvZ3JhcGhpYyBzb3J0IG9yZGVyIGFjY29yZGluZyB0byB0aGUgY3JpdGVyaWEuCmxleC5vcmRlciA8LSBmdW5jdGlvbihpZHMpIHsKICB4IDwtIHNvbHV0aW9ucyAlPiUKICAgICAgICAgZmlsdGVyKElEICVpbiUgaWRzKSAlPiUKICAgICAgICAgc2VsZWN0KGNOYW1lcykgJT4lIAogICAgICAgICBkby5jYWxsKG9yZGVyLCAuKSAlPiUgCiAgICAgICAgIHJldgogIGlkc1t4XQp9CmBgYAoKV2UgYWxzbyBuZWVkIGEgZnVuY3Rpb24gdGhhdCBjb21wYXJlcyB0d28gc29sdXRpb25zIChzcGVjaWZpZWQgYnkgSURzKSBhbmQgZGV0ZXJtaW5lcyB3aGV0aGVyIHRoZSBmaXJzdCBzb2x1dGlvbiBkb21pbmF0ZXMgdGhlIHNlY29uZCBvbmUuIE5vdGUgdGhhdCBpZiB0aGUgZmlyc3Qgc29sdXRpb24gcHJlY2VkZXMgdGhlIHNlY29uZCBzb2x1dGlvbiBpbiBvdXIgCgpgYGB7cn0KZG9taW5hdGVzIDwtIGZ1bmN0aW9uKGZpcnN0LCBzZWNvbmQpIHsKICBmIDwtIHNvbHV0aW9uc1tmaXJzdCArIDEsIF0gJT4lIHNlbGVjdChjTmFtZXMpICU+JSBhcy5udW1lcmljKCkKICBzIDwtIHNvbHV0aW9uc1tzZWNvbmQgKyAxLCBjTmFtZXNdICU+JSBhcy5udW1lcmljKCkKICBhbGwoZiA+PSBzKSAmJiBhbnkoZiA+IHMpCn0KYGBgCgojIyBGaW5kIHRoZSBQYXJldG8gZWZmaWNpZW50IHNvbHV0aW9ucwoKT3VyICJ0by1kbyIgbGlzdCB3aWxsIGJlIHRoZSBmdWxsIHNldCBvZiBJRHMsIGluIHNvcnRlZCBvcmRlciwgYWxvbmcgd2l0aCB0aGVpciBkb21pbmF0aW9uIGZsYWdzICh0byBzYXZlIHNvbWUgbG9vay11cHMpLgoKYGBge3J9CnN5c3RlbS50aW1lKHsKICB0ZW1wIDwtIGxleC5v