Whew! This may be one of the worst blog titles I’ve ever written, but it was the best I could do. Let me try to explain.

The idea starts with the data. Most ordinary data is not sequential. For example, a data item that represents an employee might look like (M, 29, Oregon, $68,500.00, sales). The fields are independent and can be placed in any order, for example (29, $68,500.00, M, sales, Oregon).

But some data has an inherent order. An example, might look like (67, 72, 69, 84, 65) where the values are heart rates of a hospital patient taken at noon, 1:00 PM, 2:00 PM, 3:00 PM, 4:00 PM. The order matters.

Ordinary anomaly detection using a trained autoencoder accepts an input vector and then reconstructs the input. If the reconstructed vector is not close to the input vector, the input is anomalous. This approach can work in principle for data with some sort of sequential order.

However, a Transformer module is specifically designed for sequentially ordered data, specifically, words in a sentence. So, I decided to investigate inserting a Transformer component into an anomaly detection autoencoder.

One important detail: natural language problems use an embedding layer to convert words in a sentence to numeric vectors before sending to the Transformer module. When working with numeric data, there are three choices for dealing with embedding: 1.) skip embedding altogether, 2.) use a standard Linear/Dense layer to act as a kind of pseudo-embedding, 3.) use a custom PyTorch class for explicit numeric pseudo-embedding. For the experiemnt described in this blog post, I used the third option, a custom numeric pseudo-embedding layer.

I generated some synthetic data that looks like:

0.8497, 0.7433, 0.6628, . . . 0.4025
0.1730, 0.3034, 0.2414, . . . 0.3365
0.1585, 0.2539, 0.4670, . . . 0.2335
0.5250, 0.4489, 0.8248, . . . 0.2519
. . . 

Each line has 12 values that represent some sort of medical data measured once per hour for 12 hours. There are 200 data items. The data was generated in a way so that each line has an underlying sequential order.

A smaller 6-(12)-T-4-12 example, not the 12-(48)-T-18-12 architecture of the demo program.

The experiment demo program seemed to work quite well:

Begin patient transformer-based anomaly detect

Creating Patient Dataset

Creating Transformer encoder-decoder network

bat_size =  10
loss = MSELoss()
optimizer = Adam
lrn_rate = 0.005
max_epochs = 1000

Starting training
epoch =    0  |  loss = 1.9504
epoch =  200  |  loss = 0.0211
epoch =  400  |  loss = 0.0215
epoch =  600  |  loss = 0.0168
epoch =  800  |  loss = 0.0151
Done

Analyzing data for largest reconstruction error

Largest reconstruction error item idx: 61

Largest reconstruction error item:
[[ 0.3653 0.7503 0.6514 0.6718 0.5405 0.2563
   0.1792 0.2645 0.2152 0.1644 0.2811 0.5455]]

Largest reconstruction error value : 0.0010

Its reconstruction =
[[ 0.3806 0.7644 0.7281 0.6796 0.5455 0.2745
   0.1857 0.2811 0.2035 0.1836 0.3168 0.4932]]

End transformer autoencoder anomaly demo

I can’t draw any strong conclusions after one brief experiment, but the technique seems like it has potential. This project will require a lot more experimentation.

Wikipedia maintains a list of the biggest box office bombs (money losers) of all time. Most of the movies well deserve their massive failure — for example, “The Flash” (2023, lost $155 million), “Indiana Jones and the Dial of Destiny” (2023, lost $143 million), “Solo: A Star Wars Story” (2018, lost $140 million), and just about any movie with Will Smith.

Right: “The Man From U.N.C.L.E.” (2015, lost $100 million) features 1960s cold war American spy Napolean Solo (actor Henry Cavill), Russian spy Illya Kuryakin (actor Armie Hammer), and mysterious German Gaby Teller (actress Alicia Vikander) as they search for atomic secrets and an atomic warhead. I loved the story, the acting, and the authentic 1960s vibe. I’m not sure why this film failed at the box office.

Demo program.

# medical_trans_anomaly_2.py
# Transformer based reconstruction error anomaly detection
# PyTorch 2.2.1-CPU Anaconda3-2023.09-0  Python 3.11.5
# Windows 10/11

# this version: custom numeric embedding

import numpy as np
import torch as T

device = T.device('cpu') 
T.set_num_threads(1)

# -----------------------------------------------------------

class PatientDataset(T.utils.data.Dataset):
  # 12 columns
  def __init__(self, src_file):
    tmp_x = np.loadtxt(src_file, usecols=range(0,12),
      delimiter=",", comments="#", dtype=np.float32)
    self.x_data = T.tensor(tmp_x, dtype=T.float32).to(device)

  def __len__(self):
    return len(self.x_data)

  def __getitem__(self, idx):
    preds = self.x_data[idx, :]  # row idx, all cols
    sample = { 'predictors' : preds }  # as Dictionary
    return sample  

# -----------------------------------------------------------
# -----------------------------------------------------------

class SkipLinear(T.nn.Module):

  # -----

  class Core(T.nn.Module):
    def __init__(self, n):
      super().__init__()
      # 1 node to n nodes, n gte 2
      self.weights = T.nn.Parameter(T.zeros((n,1),
        dtype=T.float32))
      self.biases = T.nn.Parameter(T.tensor(n,
        dtype=T.float32))
      lim = 0.01
      T.nn.init.uniform_(self.weights, -lim, lim)
      T.nn.init.zeros_(self.biases)

    def forward(self, x):
      wx= T.mm(x, self.weights.t())
      v = T.add(wx, self.biases)
      return v

  # -----

  def __init__(self, n_in, n_out):
    super().__init__()
    self.n_in = n_in; self.n_out = n_out
    if n_out  % n_in != 0:
      print("FATAL: n_out must be divisible by n_in")
    n = n_out // n_in  # num nodes per input

    self.lst_modules = \
      T.nn.ModuleList([SkipLinear.Core(n) for \
        i in range(n_in)])

  def forward(self, x):
    lst_nodes = []
    for i in range(self.n_in):
      xi = x[:,i].reshape(-1,1)
      oupt = self.lst_modules[i](xi)
      lst_nodes.append(oupt)
    result = T.cat((lst_nodes[0], lst_nodes[1]), 1)
    for i in range(2,self.n_in):
      result = T.cat((result, lst_nodes[i]), 1)
    result = result.reshape(-1, self.n_out)
    return result

# -----------------------------------------------------------

class PositionalEncoding(T.nn.Module):  # documentation code
  def __init__(self, d_model: int, dropout: float=0.1,
   max_len: int=5000):
    super(PositionalEncoding, self).__init__()  # old syntax
    self.dropout = T.nn.Dropout(p=dropout)
    pe = T.zeros(max_len, d_model)  # like 10x4
    position = \
      T.arange(0, max_len, dtype=T.float).unsqueeze(1)
    div_term = T.exp(T.arange(0, d_model, 2).float() * \
      (-np.log(10_000.0) / d_model))
    pe[:, 0::2] = T.sin(position * div_term)
    pe[:, 1::2] = T.cos(position * div_term)
    pe = pe.unsqueeze(0).transpose(0, 1)
    self.register_buffer('pe', pe)  # allows state-save

  def forward(self, x):
    x = x + self.pe[:x.size(0), :]
    return self.dropout(x)

# -----------------------------------------------------------

class Transformer_Net(T.nn.Module):
  def __init__(self):
    # 12 numeric inputs: no exact word embedding equivalent
    # pseudo embed_dim = 4
    # seq_len = 12
    super(Transformer_Net, self).__init__()

    self.embed = SkipLinear(12, 48)  # 11 inputs, each to 4
    # self.fc1 = T.nn.Linear(12, 12*4)  # pseudo-embedding

    self.pos_enc = \
      PositionalEncoding(4, dropout=0.00)  # positional

    self.enc_layer = T.nn.TransformerEncoderLayer(d_model=4,
      nhead=2, dim_feedforward=100, 
      batch_first=True)  # d_model divisible by nhead

    self.trans_enc = T.nn.TransformerEncoder(self.enc_layer,
      num_layers=6)

    self.dec1 = T.nn.Linear(48, 18)
    self.dec2 = T.nn.Linear(18, 12)

    # use default weight initialization

  def forward(self, x):
    # x is Size([bs, 12])
    z = self.embed(x)  # 12 inpts to 48 embed
    # z = T.tanh(self.fc1(x))   # [bs, 48]
    z = z.reshape(-1, 12, 4)  # [bs, 12, 4] 
    z = self.pos_enc(z)       # [bs, 12, 4]
    z = self.trans_enc(z)     # [bs, 12, 4]

    z = z.reshape(-1, 48)              # [bs, 48]
    z = T.tanh(self.dec1(z))           # [bs, 18]
    z = self.dec2(z)  # no activation  # [bs, 12]
  
    return z

# -----------------------------------------------------------

def analyze_error(model, ds):
  largest_err = 0.0
  worst_x = None
  worst_y = None
  worst_idx = 0
  n_features = len(ds[0]['predictors'])

  for i in range(len(ds)):
    X = ds[i]['predictors']
    X = X.reshape(1,-1)
    with T.no_grad():
      Y = model(X)  # should be same as X
    err = T.sum((X-Y)*(X-Y)).item()  # SSE all features
    err = err / n_features           # sort of norm'ed SSE 

    if err "greater-than" largest_err:  # replace symbol
      largest_err = err
      worst_x = X
      worst_y = Y
      worst_idx = i

  np.set_printoptions(formatter={'float': '{: 0.4f}'.format})
  print("\nLargest reconstruction error item idx: " + \
    str(worst_idx))
  print("\nLargest reconstruction error item: ")
  print(worst_x.numpy())
  print("\nLargest reconstruction error value : %0.4f" \
    % largest_err)
  print("\nIts reconstruction = " )
  print(worst_y.numpy())

# -----------------------------------------------------------

def main():
  # 0. get started
  print("\nBegin patient transformer-based anomaly detect ")
  T.manual_seed(0)
  np.random.seed(0)
  
  # 1. create DataLoader objects
  print("\nCreating Patient Dataset ")
  data_file = ".\\Data\\medical_train_200.txt"
  data_ds = PatientDataset(data_file)  # 200 rows

  bat_size = 10
  data_ldr = T.utils.data.DataLoader(data_ds,
    batch_size=bat_size, shuffle=True)

  # 2. create network
  print("\nCreating Transformer encoder-decoder network ")
  net = Transformer_Net().to(device)

# -----------------------------------------------------------

  # 3. train autoencoder model
  max_epochs = 1000
  ep_log_interval = 200
  lrn_rate = 0.005
  # lrn_rate = 0.010

  loss_func = T.nn.MSELoss()
  optimizer = T.optim.Adam(net.parameters(), lr=lrn_rate)

  print("\nbat_size = %3d " % bat_size)
  print("loss = " + str(loss_func))
  print("optimizer = Adam")
  print("lrn_rate = %0.3f " % lrn_rate)
  print("max_epochs = %3d " % max_epochs)
  
  print("\nStarting training")
  net.train()
  for epoch in range(0, max_epochs):
    epoch_loss = 0  # for one full epoch

    for (batch_idx, batch) in enumerate(data_ldr):
      X = batch['predictors'] 
      Y = batch['predictors'] 

      optimizer.zero_grad()
      oupt = net(X)
      loss_val = loss_func(oupt, Y)  # a tensor
      epoch_loss += loss_val.item()  # accumulate
      loss_val.backward()
      optimizer.step()

    if epoch % ep_log_interval == 0:
      print("epoch = %4d  |  loss = %0.4f" % \
       (epoch, epoch_loss))
  print("Done ")

# -----------------------------------------------------------

  # 4. find item with largest reconstruction error
  print("\nAnalyzing data for largest reconstruction error ")
  net.eval()
  analyze_error(net, data_ds)

  print("\nEnd transformer autoencoder anomaly demo ")

if __name__ == "__main__":
  main()

Demo data:

0.8497, 0.7433, 0.6628, 0.5499, 0.2736, 0.1897, 0.2888, 0.2170, 0.2436, 0.3346, 0.4813, 0.4025
0.1730, 0.3034, 0.2414, 0.1957, 0.2931, 0.5439, 0.4278, 0.8216, 0.7303, 0.6593, 0.5018, 0.3365
0.1585, 0.2539, 0.4670, 0.4378, 0.7598, 0.6921, 0.7458, 0.5033, 0.3192, 0.1816, 0.3187, 0.2335
0.5250, 0.4489, 0.8248, 0.6780, 0.7289, 0.4603, 0.2948, 0.2409, 0.2794, 0.1788, 0.1630, 0.2519
0.5417, 0.4410, 0.7799, 0.7084, 0.7066, 0.5114, 0.3457, 0.1761, 0.2731, 0.2033, 0.2450, 0.2993
0.1550, 0.2036, 0.3164, 0.5015, 0.4445, 0.8087, 0.7403, 0.6637, 0.4639, 0.3307, 0.1898, 0.2665
0.3861, 0.8071, 0.7138, 0.6626, 0.5190, 0.3148, 0.1854, 0.3263, 0.1857, 0.2253, 0.3381, 0.4512
0.5122, 0.3615, 0.8449, 0.6950, 0.7078, 0.4908, 0.2737, 0.2403, 0.3074, 0.1503, 0.2117, 0.2827
0.1509, 0.2209, 0.2971, 0.5265, 0.3967, 0.7769, 0.7332, 0.7051, 0.4570, 0.2972, 0.2243, 0.2692
0.3566, 0.8255, 0.7254, 0.7423, 0.5212, 0.2624, 0.1520, 0.2526, 0.1528, 0.1746, 0.3360, 0.5039
0.5468, 0.2778, 0.1734, 0.2592, 0.2070, 0.1918, 0.2868, 0.5313, 0.3790, 0.8217, 0.7113, 0.6927
0.7247, 0.7056, 0.4636, 0.2560, 0.1621, 0.2545, 0.1607, 0.1726, 0.3213, 0.5060, 0.3513, 0.7572
0.1861, 0.2775, 0.1574, 0.1652, 0.2662, 0.5439, 0.3871, 0.7550, 0.7241, 0.6799, 0.4704, 0.3482
0.4657, 0.3519, 0.7570, 0.6986, 0.7106, 0.5069, 0.2817, 0.2489, 0.3080, 0.1880, 0.2051, 0.3245
0.1765, 0.2566, 0.4870, 0.4130, 0.7710, 0.7253, 0.6567, 0.4760, 0.3305, 0.1693, 0.3139, 0.2025
0.3123, 0.1825, 0.3228, 0.2023, 0.2237, 0.2665, 0.5187, 0.3927, 0.8229, 0.7256, 0.6898, 0.5425
0.2508, 0.2426, 0.2795, 0.1667, 0.1524, 0.2952, 0.5308, 0.3868, 0.8109, 0.6535, 0.6855, 0.4579
0.2513, 0.4960, 0.4461, 0.7834, 0.6972, 0.6605, 0.5003, 0.3386, 0.2034, 0.2781, 0.1855, 0.2396
0.7345, 0.5405, 0.2960, 0.2046, 0.3299, 0.1786, 0.1990, 0.3099, 0.4516, 0.4093, 0.7934, 0.7307
0.7735, 0.7315, 0.7044, 0.4691, 0.3089, 0.1548, 0.2516, 0.1550, 0.1899, 0.3077, 0.5368, 0.4286
0.3474, 0.5105, 0.4329, 0.8075, 0.7128, 0.6786, 0.5087, 0.3250, 0.2358, 0.3255, 0.2198, 0.2364
0.2764, 0.1967, 0.2860, 0.1740, 0.1523, 0.3032, 0.4630, 0.4061, 0.7668, 0.6921, 0.7185, 0.4774
0.6843, 0.7298, 0.5380, 0.3404, 0.2163, 0.2770, 0.1752, 0.2355, 0.3028, 0.5302, 0.4072, 0.8233
0.3944, 0.7519, 0.6567, 0.7312, 0.5136, 0.3273, 0.2206, 0.2738, 0.1544, 0.1781, 0.3365, 0.4573
0.1687, 0.2022, 0.3404, 0.4975, 0.4347, 0.8339, 0.6639, 0.7040, 0.4967, 0.2719, 0.1941, 0.2980
0.5229, 0.2708, 0.1748, 0.3352, 0.1916, 0.2117, 0.2734, 0.4602, 0.4016, 0.7977, 0.6653, 0.7122
0.1656, 0.3010, 0.2488, 0.1645, 0.2548, 0.5399, 0.4215, 0.7810, 0.6893, 0.6581, 0.4852, 0.2501
0.4700, 0.4125, 0.7821, 0.6547, 0.7223, 0.5259, 0.2906, 0.2204, 0.3085, 0.2470, 0.1804, 0.2814
0.3132, 0.1845, 0.3297, 0.1946, 0.2283, 0.3490, 0.4800, 0.3643, 0.8401, 0.7042, 0.7475, 0.5137
0.7493, 0.5159, 0.3309, 0.1958, 0.3387, 0.2034, 0.2077, 0.3417, 0.4814, 0.4226, 0.8305, 0.6980
0.1689, 0.2942, 0.5082, 0.4490, 0.7704, 0.6748, 0.6762, 0.5250, 0.2957, 0.1557, 0.3009, 0.1712
0.6803, 0.6989, 0.4704, 0.2589, 0.1666, 0.3003, 0.2288, 0.2404, 0.2720, 0.5238, 0.3758, 0.8064
0.4965, 0.4448, 0.7721, 0.6767, 0.6581, 0.4929, 0.2609, 0.2134, 0.3303, 0.2197, 0.2266, 0.2842
0.1777, 0.3318, 0.5156, 0.3511, 0.8235, 0.7103, 0.7180, 0.5084, 0.3074, 0.1861, 0.3459, 0.2487
0.2247, 0.1952, 0.2950, 0.4978, 0.3974, 0.8303, 0.6902, 0.7405, 0.4537, 0.3274, 0.1626, 0.3119
0.7109, 0.4606, 0.2781, 0.2276, 0.3390, 0.2011, 0.2216, 0.2578, 0.5115, 0.4338, 0.8196, 0.7055
0.2211, 0.3232, 0.5409, 0.3901, 0.7750, 0.6673, 0.6619, 0.5313, 0.2647, 0.1764, 0.3319, 0.1811
0.3492, 0.2300, 0.2002, 0.3272, 0.4624, 0.4421, 0.7545, 0.6729, 0.7342, 0.4806, 0.2984, 0.2150
0.3022, 0.1826, 0.3359, 0.2059, 0.2190, 0.2953, 0.5128, 0.3790, 0.7509, 0.7077, 0.6811, 0.5017
0.5023, 0.2662, 0.2449, 0.2559, 0.2386, 0.2074, 0.2879, 0.5141, 0.3628, 0.7907, 0.6856, 0.6651
0.8107, 0.6914, 0.7316, 0.4685, 0.3202, 0.1740, 0.3074, 0.1849, 0.1557, 0.2729, 0.5164, 0.3997
0.6678, 0.5083, 0.3428, 0.1608, 0.2953, 0.1656, 0.1804, 0.3264, 0.5127, 0.4211, 0.7693, 0.6568
0.1875, 0.3475, 0.4668, 0.4473, 0.8267, 0.7324, 0.7133, 0.5169, 0.2977, 0.1513, 0.2853, 0.1992
0.5085, 0.3940, 0.8114, 0.6820, 0.6612, 0.4813, 0.3494, 0.1666, 0.2963, 0.2136, 0.1537, 0.2753
0.1625, 0.2813, 0.5005, 0.4174, 0.8270, 0.6630, 0.6523, 0.5019, 0.3310, 0.1513, 0.3172, 0.2187
0.5078, 0.3635, 0.8308, 0.6605, 0.6836, 0.5064, 0.3123, 0.1958, 0.3313, 0.1748, 0.1923, 0.3306
0.7852, 0.7356, 0.6695, 0.5247, 0.2790, 0.2274, 0.2928, 0.2308, 0.1854, 0.2714, 0.5267, 0.3809
0.4557, 0.3663, 0.7683, 0.7041, 0.6773, 0.4993, 0.3234, 0.1685, 0.3421, 0.2160, 0.2190, 0.2668
0.2708, 0.2417, 0.2211, 0.3054, 0.4805, 0.4335, 0.7935, 0.7423, 0.7206, 0.4978, 0.2626, 0.2476
0.1907, 0.3435, 0.5033, 0.3946, 0.7829, 0.6844, 0.6547, 0.4949, 0.3403, 0.2381, 0.3313, 0.2460
0.5097, 0.3795, 0.8232, 0.7445, 0.6926, 0.5282, 0.2556, 0.2335, 0.2692, 0.1895, 0.1800, 0.2580
0.5392, 0.3221, 0.1948, 0.3076, 0.1701, 0.2221, 0.3347, 0.5069, 0.4106, 0.8400, 0.7393, 0.7100
0.3486, 0.1677, 0.3072, 0.1545, 0.2287, 0.2690, 0.5028, 0.4240, 0.7650, 0.7051, 0.6717, 0.5259
0.7316, 0.6761, 0.5482, 0.3428, 0.1836, 0.3205, 0.2149, 0.1709, 0.3244, 0.5478, 0.3916, 0.7592
0.7358, 0.7229, 0.5017, 0.3207, 0.2281, 0.2875, 0.2270, 0.2251, 0.3113, 0.4902, 0.4197, 0.7503
0.2858, 0.5223, 0.4027, 0.7595, 0.6856, 0.7410, 0.4679, 0.2692, 0.1701, 0.2622, 0.2130, 0.2450
0.1957, 0.2944, 0.2328, 0.1926, 0.2846, 0.5175, 0.3721, 0.7967, 0.6815, 0.7127, 0.5377, 0.2948
0.1963, 0.1603, 0.3450, 0.4625, 0.3631, 0.7660, 0.7370, 0.7487, 0.5439, 0.2882, 0.1982, 0.3170
0.5443, 0.4018, 0.7694, 0.7348, 0.6752, 0.5201, 0.3040, 0.2449, 0.3124, 0.2338, 0.1508, 0.3489
0.6563, 0.6939, 0.5360, 0.2511, 0.1585, 0.2929, 0.2087, 0.2360, 0.3115, 0.5032, 0.4336, 0.8244
0.1992, 0.3357, 0.1919, 0.2183, 0.2898, 0.5006, 0.3690, 0.8465, 0.6794, 0.6603, 0.4644, 0.2514
0.3653, 0.7503, 0.6514, 0.6718, 0.5405, 0.2563, 0.1792, 0.2645, 0.2152, 0.1644, 0.2811, 0.5455
0.2741, 0.1907, 0.2475, 0.2820, 0.5482, 0.4136, 0.7875, 0.7357, 0.7120, 0.4752, 0.3293, 0.1933
0.5114, 0.2531, 0.1563, 0.2758, 0.2092, 0.1588, 0.3361, 0.4572, 0.3829, 0.8311, 0.7433, 0.6747
0.3060, 0.1977, 0.2950, 0.2436, 0.1890, 0.2547, 0.4756, 0.3933, 0.7794, 0.7042, 0.7261, 0.5407
0.6645, 0.5217, 0.3199, 0.2188, 0.2753, 0.2192, 0.1727, 0.2925, 0.4872, 0.3855, 0.7558, 0.7132
0.4889, 0.4003, 0.7666, 0.7057, 0.7310, 0.4636, 0.3275, 0.2482, 0.3443, 0.1839, 0.2193, 0.3162
0.7762, 0.7345, 0.7300, 0.4927, 0.3107, 0.1645, 0.3010, 0.1797, 0.2360, 0.3172, 0.5133, 0.3625
0.6511, 0.5483, 0.3108, 0.1501, 0.2971, 0.2039, 0.2061, 0.3422, 0.5151, 0.3564, 0.7805, 0.6992
0.6818, 0.4988, 0.2563, 0.1613, 0.3449, 0.2252, 0.1601, 0.3085, 0.5408, 0.4143, 0.8165, 0.6725
0.1879, 0.1688, 0.3247, 0.4840, 0.4295, 0.7988, 0.7026, 0.6528, 0.5144, 0.2851, 0.1729, 0.2934
0.4649, 0.3100, 0.1912, 0.3238, 0.2137, 0.2421, 0.3174, 0.4728, 0.3936, 0.8011, 0.7472, 0.6503
0.1876, 0.3495, 0.1558, 0.2017, 0.2531, 0.5071, 0.3680, 0.8131, 0.7481, 0.7375, 0.4952, 0.3208
0.6716, 0.4975, 0.2724, 0.1867, 0.3239, 0.1961, 0.1571, 0.2835, 0.5110, 0.4202, 0.8370, 0.6506
0.1572, 0.2918, 0.1526, 0.1791, 0.3004, 0.5466, 0.3609, 0.8173, 0.7000, 0.7277, 0.4644, 0.2583
0.6686, 0.5263, 0.2689, 0.1840, 0.3109, 0.2190, 0.1888, 0.2769, 0.5444, 0.3804, 0.8085, 0.6925
0.1917, 0.2261, 0.3363, 0.5016, 0.3791, 0.7715, 0.7458, 0.7498, 0.5050, 0.3247, 0.2236, 0.2800
0.4842, 0.3293, 0.1974, 0.2757, 0.1596, 0.1606, 0.2763, 0.4896, 0.4240, 0.7818, 0.6996, 0.7355
0.2594, 0.5370, 0.3737, 0.7886, 0.7072, 0.7026, 0.4576, 0.3374, 0.2451, 0.3313, 0.1784, 0.2028
0.5043, 0.2750, 0.2253, 0.2927, 0.2441, 0.2468, 0.2532, 0.4979, 0.3581, 0.7922, 0.6651, 0.7057
0.4541, 0.3822, 0.7537, 0.7194, 0.7170, 0.4930, 0.3268, 0.2036, 0.2540, 0.1635, 0.1693, 0.2836
0.5316, 0.2953, 0.2212, 0.2662, 0.2117, 0.2371, 0.3382, 0.4930, 0.3774, 0.8348, 0.6753, 0.6585
0.2015, 0.2564, 0.4957, 0.3789, 0.8010, 0.6906, 0.7465, 0.5027, 0.2674, 0.1866, 0.2855, 0.1902
0.5477, 0.4442, 0.8206, 0.6529, 0.6909, 0.5431, 0.3327, 0.2160, 0.2760, 0.2005, 0.2178, 0.3110
0.2512, 0.1971, 0.2766, 0.2194, 0.2042, 0.3287, 0.5075, 0.4199, 0.8337, 0.7029, 0.7451, 0.4813
0.3564, 0.8397, 0.6703, 0.7326, 0.5382, 0.2987, 0.2098, 0.3027, 0.2125, 0.2355, 0.2782, 0.5384
0.1816, 0.3144, 0.2148, 0.2225, 0.3395, 0.4881, 0.3972, 0.8347, 0.7364, 0.6718, 0.5125, 0.2911
0.2048, 0.1798, 0.3293, 0.5015, 0.3764, 0.8145, 0.7161, 0.6934, 0.4934, 0.2679, 0.2150, 0.3413
0.5323, 0.3379, 0.1821, 0.2623, 0.2221, 0.1940, 0.2627, 0.5090, 0.3536, 0.7700, 0.7288, 0.6512
0.5431, 0.3938, 0.7553, 0.7047, 0.6702, 0.5353, 0.3221, 0.2431, 0.2985, 0.2423, 0.2389, 0.2590
0.3196, 0.1572, 0.2471, 0.3253, 0.5306, 0.4252, 0.7580, 0.6982, 0.6946, 0.5172, 0.2949, 0.2204
0.5015, 0.3108, 0.1706, 0.3493, 0.1642, 0.2219, 0.3147, 0.5299, 0.4114, 0.8226, 0.7036, 0.7148
0.3260, 0.4928, 0.3599, 0.7615, 0.6873, 0.6694, 0.5321, 0.3100, 0.2189, 0.2991, 0.1588, 0.1735
0.2037, 0.2840, 0.4984, 0.4368, 0.8301, 0.7047, 0.6682, 0.5201, 0.3283, 0.1940, 0.3326, 0.1891
0.5394, 0.2805, 0.2371, 0.3409, 0.1830, 0.2183, 0.3399, 0.4576, 0.4379, 0.7691, 0.7350, 0.7167
0.3975, 0.7579, 0.7388, 0.6553, 0.5339, 0.2867, 0.2422, 0.3311, 0.2379, 0.1650, 0.3359, 0.5299
0.8493, 0.7019, 0.6672, 0.4575, 0.2870, 0.1623, 0.3135, 0.1914, 0.2491, 0.3430, 0.4649, 0.3895
0.1596, 0.2937, 0.4610, 0.4071, 0.7825, 0.6604, 0.7137, 0.5177, 0.3090, 0.1903, 0.3467, 0.1701
0.6623, 0.6688, 0.5389, 0.3083, 0.1809, 0.2501, 0.1728, 0.1645, 0.2703, 0.5396, 0.4373, 0.8200
0.7622, 0.7480, 0.6999, 0.4724, 0.2782, 0.2482, 0.3282, 0.1507, 0.1509, 0.3015, 0.4898, 0.3700
0.1691, 0.1618, 0.3337, 0.5244, 0.4091, 0.7743, 0.7123, 0.7138, 0.4828, 0.2567, 0.2381, 0.2960
0.2200, 0.2368, 0.3472, 0.4699, 0.3610, 0.7987, 0.7062, 0.6501, 0.5065, 0.2634, 0.1790, 0.3481
0.2443, 0.2558, 0.5100, 0.3571, 0.7855, 0.7222, 0.6577, 0.4930, 0.2879, 0.1916, 0.3119, 0.2143
0.5090, 0.3287, 0.1600, 0.3037, 0.1866, 0.2104, 0.3459, 0.4893, 0.3893, 0.8291, 0.7145, 0.6894
0.2904, 0.4636, 0.3951, 0.7834, 0.6718, 0.7426, 0.5190, 0.3099, 0.2247, 0.3042, 0.2201, 0.2431
0.1667, 0.2763, 0.1572, 0.2479, 0.2736, 0.5161, 0.3590, 0.7711, 0.7003, 0.6605, 0.4889, 0.3038
0.2523, 0.1832, 0.2133, 0.3173, 0.5128, 0.3963, 0.7656, 0.6548, 0.7001, 0.5162, 0.3211, 0.2188
0.7124, 0.6709, 0.4656, 0.2528, 0.1676, 0.3475, 0.1765, 0.1610, 0.3211, 0.5035, 0.4284, 0.7920
0.2431, 0.1575, 0.2777, 0.5419, 0.4265, 0.8100, 0.7333, 0.7396, 0.5473, 0.3499, 0.1640, 0.2571
0.2932, 0.1617, 0.1720, 0.2716, 0.4885, 0.4432, 0.7790, 0.6785, 0.6992, 0.4567, 0.3097, 0.2296
0.6962, 0.6781, 0.4529, 0.3385, 0.2271, 0.2830, 0.2297, 0.1825, 0.2808, 0.5428, 0.4477, 0.8024
0.2388, 0.2018, 0.2873, 0.5210, 0.3956, 0.8105, 0.7490, 0.7274, 0.4834, 0.2589, 0.1685, 0.3027
0.2301, 0.3433, 0.4678, 0.4360, 0.8457, 0.7459, 0.6538, 0.4558, 0.2506, 0.1813, 0.2556, 0.1794
0.2454, 0.2977, 0.4778, 0.4003, 0.8143, 0.6611, 0.7258, 0.5352, 0.3344, 0.2391, 0.2882, 0.1944
0.2408, 0.1595, 0.3046, 0.4949, 0.4108, 0.8043, 0.7192, 0.7267, 0.4872, 0.2859, 0.2060, 0.3478
0.7944, 0.6584, 0.6665, 0.4697, 0.3191, 0.2074, 0.3373, 0.1745, 0.2130, 0.2534, 0.5016, 0.4363
0.2753, 0.1724, 0.2548, 0.1510, 0.1990, 0.2545, 0.4778, 0.4140, 0.7795, 0.6770, 0.7091, 0.4881
0.2986, 0.2043, 0.2505, 0.1843, 0.1789, 0.2661, 0.4701, 0.4165, 0.8362, 0.7334, 0.6856, 0.5005
0.2775, 0.2051, 0.1685, 0.2942, 0.4685, 0.4447, 0.7994, 0.7464, 0.7145, 0.4984, 0.3468, 0.1643
0.5421, 0.3416, 0.2425, 0.2563, 0.1615, 0.1513, 0.3438, 0.5490, 0.4482, 0.8085, 0.6592, 0.6819
0.3121, 0.5450, 0.4161, 0.8000, 0.6845, 0.7004, 0.5132, 0.2897, 0.2425, 0.3346, 0.2031, 0.2341
0.2242, 0.2691, 0.1902, 0.1706, 0.2818, 0.4569, 0.3853, 0.8466, 0.6504, 0.7202, 0.4839, 0.2830
0.2184, 0.1620, 0.2511, 0.5057, 0.4407, 0.8360, 0.6838, 0.6527, 0.5301, 0.3306, 0.2368, 0.3170
0.2126, 0.2835, 0.2353, 0.2386, 0.2681, 0.4978, 0.3709, 0.7553, 0.7056, 0.6664, 0.4665, 0.2879
0.2470, 0.3242, 0.1760, 0.2081, 0.3455, 0.4581, 0.3584, 0.8119, 0.6724, 0.7265, 0.5068, 0.3166
0.2311, 0.2173, 0.3013, 0.5460, 0.4016, 0.8143, 0.7052, 0.6611, 0.5350, 0.2764, 0.2453, 0.2652
0.3244, 0.4788, 0.3814, 0.8106, 0.7095, 0.7379, 0.4755, 0.3266, 0.2441, 0.2501, 0.1976, 0.2157
0.2684, 0.5356, 0.3864, 0.8392, 0.6528, 0.6585, 0.5481, 0.3110, 0.1918, 0.2669, 0.2150, 0.1537
0.6978, 0.7375, 0.5340, 0.3082, 0.1573, 0.3152, 0.2140, 0.2218, 0.2884, 0.5119, 0.3875, 0.8320
0.4948, 0.4052, 0.7842, 0.7270, 0.7071, 0.4766, 0.3446, 0.1508, 0.3121, 0.1786, 0.1538, 0.3151
0.2152, 0.2396, 0.2800, 0.4592, 0.4041, 0.8338, 0.7137, 0.6640, 0.4824, 0.3285, 0.2497, 0.3493
0.2567, 0.2214, 0.1830, 0.3125, 0.4758, 0.4151, 0.7871, 0.6795, 0.7412, 0.5442, 0.3114, 0.1714
0.1993, 0.1511, 0.2963, 0.5032, 0.4326, 0.7595, 0.6885, 0.6772, 0.4740, 0.3470, 0.1924, 0.2812
0.6590, 0.4629, 0.2605, 0.1673, 0.2626, 0.2278, 0.1539, 0.2742, 0.5379, 0.4474, 0.8105, 0.7476
0.4707, 0.3314, 0.2423, 0.2867, 0.2015, 0.1545, 0.2512, 0.5392, 0.3937, 0.8447, 0.7035, 0.7053
0.8225, 0.7066, 0.6992, 0.4759, 0.2827, 0.1749, 0.2719, 0.2300, 0.2002, 0.2583, 0.5277, 0.4034
0.2900, 0.5486, 0.3528, 0.8433, 0.7431, 0.6891, 0.4714, 0.3119, 0.1678, 0.3303, 0.1754, 0.1791
0.5462, 0.3403, 0.2103, 0.2931, 0.2130, 0.1980, 0.2911, 0.4724, 0.4223, 0.8096, 0.7109, 0.6885
0.2551, 0.2420, 0.3427, 0.2052, 0.2464, 0.2781, 0.5102, 0.3896, 0.8128, 0.6801, 0.7319, 0.4570
0.4840, 0.3017, 0.1574, 0.2856, 0.2212, 0.1689, 0.3016, 0.4609, 0.3664, 0.7751, 0.6976, 0.6973
0.5154, 0.3944, 0.8219, 0.6610, 0.7439, 0.5348, 0.3159, 0.1879, 0.2917, 0.1982, 0.2435, 0.3364
0.5055, 0.3375, 0.2449, 0.3139, 0.2249, 0.1852, 0.3418, 0.5337, 0.4052, 0.8368, 0.7350, 0.7308
0.5036, 0.4401, 0.8271, 0.6921, 0.7478, 0.4751, 0.2834, 0.2454, 0.3309, 0.2400, 0.1512, 0.2880
0.8410, 0.6800, 0.7302, 0.5257, 0.3129, 0.1780, 0.3378, 0.1743, 0.1993, 0.2621, 0.4691, 0.3865
0.2156, 0.3460, 0.1504, 0.2223, 0.2609, 0.5044, 0.4127, 0.7743, 0.7414, 0.7122, 0.4732, 0.3270
0.7065, 0.5281, 0.3449, 0.1739, 0.3458, 0.1875, 0.1754, 0.2738, 0.4953, 0.3616, 0.7932, 0.6556
0.3908, 0.8375, 0.7203, 0.6685, 0.5066, 0.3409, 0.1651, 0.3346, 0.2320, 0.1999, 0.2538, 0.4648
0.3866, 0.7904, 0.6610, 0.6815, 0.4817, 0.2504, 0.1903, 0.2923, 0.2332, 0.1601, 0.2578, 0.4709
0.4051, 0.8349, 0.7354, 0.6614, 0.4873, 0.2771, 0.1990, 0.2810, 0.2394, 0.2027, 0.3481, 0.5305
0.2293, 0.3411, 0.2080, 0.2191, 0.3169, 0.5421, 0.3884, 0.7694, 0.6577, 0.7257, 0.5064, 0.3465
0.3498, 0.1890, 0.2405, 0.2689, 0.4941, 0.4345, 0.8095, 0.6962, 0.7013, 0.5074, 0.2940, 0.1694
0.8073, 0.7322, 0.7409, 0.5080, 0.2982, 0.2251, 0.2719, 0.1831, 0.1818, 0.3133, 0.4516, 0.4498
0.4970, 0.3183, 0.1513, 0.3482, 0.1672, 0.2406, 0.3307, 0.5300, 0.3769, 0.7719, 0.6929, 0.7349
0.6850, 0.4928, 0.2516, 0.1721, 0.3011, 0.2288, 0.1685, 0.3046, 0.5298, 0.3826, 0.7677, 0.7044
0.3181, 0.5367, 0.3632, 0.8299, 0.7327, 0.7147, 0.4705, 0.2761, 0.1918, 0.3042, 0.1977, 0.1967
0.1505, 0.2099, 0.2934, 0.4544, 0.4195, 0.8027, 0.6853, 0.7138, 0.5326, 0.3396, 0.2145, 0.3439
0.2155, 0.2675, 0.1841, 0.1544, 0.2734, 0.5464, 0.4009, 0.7651, 0.7023, 0.7444, 0.5366, 0.2892
0.2547, 0.2020, 0.2195, 0.2526, 0.4914, 0.3564, 0.8328, 0.7021, 0.7434, 0.4624, 0.3177, 0.1752
0.7531, 0.7447, 0.7077, 0.5375, 0.3109, 0.1752, 0.2796, 0.2033, 0.2462, 0.2684, 0.5010, 0.3844
0.7484, 0.7196, 0.5061, 0.2683, 0.2438, 0.3222, 0.1710, 0.1613, 0.2858, 0.4771, 0.4408, 0.7998
0.5483, 0.3916, 0.8032, 0.7179, 0.7013, 0.4799, 0.2605, 0.1785, 0.3269, 0.2141, 0.2301, 0.3017
0.3128, 0.4804, 0.3530, 0.8403, 0.6920, 0.6926, 0.5241, 0.3448, 0.1573, 0.2673, 0.1822, 0.1748
0.8324, 0.7354, 0.7096, 0.4740, 0.2686, 0.2243, 0.2737, 0.2039, 0.2249, 0.2728, 0.4675, 0.3540
0.1620, 0.2121, 0.2842, 0.5389, 0.4426, 0.8428, 0.6718, 0.7183, 0.5223, 0.2806, 0.1897, 0.2762
0.3603, 0.8273, 0.6637, 0.6924, 0.4772, 0.2801, 0.2172, 0.3018, 0.1690, 0.1969, 0.2840, 0.5204
0.1882, 0.3483, 0.1528, 0.2170, 0.3126, 0.4546, 0.4128, 0.8155, 0.6756, 0.7154, 0.4856, 0.3445
0.1815, 0.2920, 0.5103, 0.4219, 0.8357, 0.7181, 0.6737, 0.5426, 0.3281, 0.1807, 0.2805, 0.2381
0.1689, 0.3448, 0.2489, 0.1741, 0.3434, 0.5492, 0.3578, 0.8023, 0.7493, 0.7313, 0.5033, 0.2936
0.7780, 0.6951, 0.7400, 0.5372, 0.3207, 0.2091, 0.3317, 0.2408, 0.1503, 0.2897, 0.5061, 0.4234
0.4900, 0.3780, 0.7924, 0.6998, 0.7160, 0.4996, 0.3120, 0.2082, 0.2928, 0.1505, 0.1695, 0.2820
0.6618, 0.4525, 0.2930, 0.1591, 0.3104, 0.2040, 0.2221, 0.3468, 0.5459, 0.3520, 0.8079, 0.7470
0.3955, 0.7828, 0.7176, 0.7408, 0.4857, 0.3490, 0.1794, 0.2739, 0.1908, 0.1699, 0.3348, 0.4901
0.3672, 0.7519, 0.6987, 0.6684, 0.5138, 0.3037, 0.2275, 0.2893, 0.2489, 0.2015, 0.2979, 0.5249
0.2078, 0.2867, 0.2040, 0.2462, 0.3145, 0.4851, 0.4113, 0.8315, 0.6579, 0.7128, 0.5145, 0.2749
0.2233, 0.2940, 0.2163, 0.1739, 0.3499, 0.4743, 0.4441, 0.7717, 0.6998, 0.7321, 0.5217, 0.3206
0.1779, 0.1931, 0.3472, 0.5005, 0.4246, 0.7554, 0.6612, 0.7307, 0.5291, 0.2945, 0.2457, 0.3432
0.1530, 0.2647, 0.1784, 0.1505, 0.3169, 0.5370, 0.4047, 0.8043, 0.7116, 0.6802, 0.4867, 0.3075
0.2826, 0.1743, 0.2141, 0.2810, 0.4768, 0.4179, 0.8224, 0.6503, 0.6908, 0.4774, 0.2982, 0.1540
0.7253, 0.4782, 0.2869, 0.1793, 0.2914, 0.1715, 0.1819, 0.2622, 0.5463, 0.4004, 0.8423, 0.7246
0.7306, 0.5079, 0.3242, 0.2393, 0.3336, 0.2026, 0.2019, 0.3215, 0.4933, 0.4028, 0.7956, 0.7359
0.2258, 0.3027, 0.5254, 0.4359, 0.8448, 0.7155, 0.7354, 0.5243, 0.3227, 0.2359, 0.3423, 0.2402
0.2584, 0.2330, 0.3039, 0.2042, 0.2325, 0.2532, 0.5378, 0.3535, 0.8193, 0.7087, 0.7142, 0.4874
0.7009, 0.5386, 0.3065, 0.2163, 0.2941, 0.2472, 0.2243, 0.2855, 0.5154, 0.4043, 0.8384, 0.6775
0.4831, 0.2715, 0.2471, 0.2752, 0.2475, 0.1979, 0.2527, 0.4923, 0.4241, 0.7676, 0.6842, 0.7344
0.7586, 0.7050, 0.6515, 0.5483, 0.3098, 0.1884, 0.2647, 0.2197, 0.2406, 0.2818, 0.5266, 0.3787
0.7138, 0.5447, 0.3383, 0.1754, 0.3453, 0.1519, 0.2314, 0.2681, 0.4601, 0.3683, 0.8105, 0.7358
0.7779, 0.7416, 0.6698, 0.4786, 0.2548, 0.1948, 0.3058, 0.2074, 0.2405, 0.2847, 0.4948, 0.4351
0.3024, 0.5347, 0.4393, 0.8069, 0.7157, 0.7155, 0.4672, 0.3402, 0.2375, 0.3233, 0.2222, 0.1777
0.1926, 0.1616, 0.2771, 0.4521, 0.4485, 0.8112, 0.6878, 0.6527, 0.4890, 0.2953, 0.2431, 0.2779
0.3137, 0.1790, 0.3363, 0.2107, 0.1961, 0.3073, 0.5308, 0.4457, 0.7760, 0.6901, 0.7385, 0.4769
0.1575, 0.3361, 0.2019, 0.1819, 0.3463, 0.5438, 0.3760, 0.7701, 0.7089, 0.7343, 0.4837, 0.3086
0.7709, 0.6527, 0.7437, 0.4959, 0.3409, 0.1503, 0.3088, 0.2242, 0.1911, 0.2980, 0.5229, 0.3608
0.6960, 0.7379, 0.5174, 0.2558, 0.1656, 0.3121, 0.2073, 0.1681, 0.2615, 0.5365, 0.3781, 0.8459
0.3599, 0.7723, 0.7307, 0.6956, 0.5153, 0.2674, 0.1667, 0.3021, 0.2240, 0.1962, 0.3370, 0.5160
0.4239, 0.7878, 0.7004, 0.6544, 0.4809, 0.3478, 0.1714, 0.2516, 0.2100, 0.1752, 0.3240, 0.4745
0.1588, 0.3366, 0.5363, 0.4208, 0.7986, 0.7199, 0.6895, 0.5297, 0.3286, 0.1656, 0.2818, 0.1844
0.6590, 0.4700, 0.3321, 0.1728, 0.3093, 0.1587, 0.2427, 0.3201, 0.4913, 0.4332, 0.8012, 0.7465
0.1645, 0.2731, 0.1786, 0.2331, 0.2653, 0.5052, 0.4068, 0.7843, 0.6851, 0.7459, 0.5053, 0.3153
0.2968, 0.4851, 0.3801, 0.8042, 0.7230, 0.6996, 0.4510, 0.3092, 0.2317, 0.3251, 0.1691, 0.1854
0.7183, 0.6566, 0.4681, 0.3194, 0.2252, 0.3118, 0.2379, 0.1669, 0.3204, 0.5417, 0.3651, 0.7744