sys.stdout.flush

Here are the examples of the python api sys.stdout.flush taken from open source projects. By voting up you can indicate which examples are most useful and appropriate.

200 Examples 7

Example 101

Project: trtools
Source File: rmagic.py
View license
    @skip_doctest
    @magic_arguments()
    @argument(
        '-i', '--input', action='append',
        help='Names of input variable from shell.user_ns to be assigned to R variables of the same names after calling self.pyconverter. Multiple names can be passed separated only by commas with no whitespace.'
        )
    @argument(
        '-o', '--output', action='append',
        help='Names of variables to be pushed from rpy2 to shell.user_ns after executing cell body and applying self.Rconverter. Multiple names can be passed separated only by commas with no whitespace.'
        )
    @argument(
        '-w', '--width', type=int,
        help='Width of png plotting device sent as an argument to *png* in R.'
        )
    @argument(
        '-h', '--height', type=int,
        help='Height of png plotting device sent as an argument to *png* in R.'
        )

    @argument(
        '-d', '--dataframe', action='append',
        help='Convert these objects to data.frames and return as structured arrays.'
        )
    @argument(
        '-u', '--units', type=int,
        help='Units of png plotting device sent as an argument to *png* in R. One of ["px", "in", "cm", "mm"].'
        )
    @argument(
        '-p', '--pointsize', type=int,
        help='Pointsize of png plotting device sent as an argument to *png* in R.'
        )
    @argument(
        '-b', '--bg',
        help='Background of png plotting device sent as an argument to *png* in R.'
        )
    @argument(
        '-n', '--noreturn',
        help='Force the magic to not return anything.',
        action='store_true',
        default=False
        )
    @argument(
        'code',
        nargs='*',
        )
    @needs_local_scope
    @line_cell_magic
    def R(self, line, cell=None, local_ns=None):
        '''
        Execute code in R, and pull some of the results back into the Python namespace.

        In line mode, this will evaluate an expression and convert the returned value to a Python object.
        The return value is determined by rpy2's behaviour of returning the result of evaluating the
        final line. 

        Multiple R lines can be executed by joining them with semicolons::

            In [9]: %R X=c(1,4,5,7); sd(X); mean(X)
            Out[9]: array([ 4.25])

        As a cell, this will run a block of R code, without bringing anything back by default::

            In [10]: %%R
               ....: Y = c(2,4,3,9)
               ....: print(summary(lm(Y~X)))
               ....:

            Call:
            lm(formula = Y ~ X)

            Residuals:
                1     2     3     4
             0.88 -0.24 -2.28  1.64

            Coefficients:
                        Estimate Std. Error t value Pr(>|t|)
            (Intercept)   0.0800     2.3000   0.035    0.975
            X             1.0400     0.4822   2.157    0.164

            Residual standard error: 2.088 on 2 degrees of freedom
            Multiple R-squared: 0.6993,Adjusted R-squared: 0.549
            F-statistic: 4.651 on 1 and 2 DF,  p-value: 0.1638

        In the notebook, plots are published as the output of the cell.

        %R plot(X, Y)

        will create a scatter plot of X bs Y.

        If cell is not None and line has some R code, it is prepended to
        the R code in cell.

        Objects can be passed back and forth between rpy2 and python via the -i -o flags in line::

            In [14]: Z = np.array([1,4,5,10])

            In [15]: %R -i Z mean(Z)
            Out[15]: array([ 5.])


            In [16]: %R -o W W=Z*mean(Z)
            Out[16]: array([  5.,  20.,  25.,  50.])

            In [17]: W
            Out[17]: array([  5.,  20.,  25.,  50.])

        The return value is determined by these rules:

        * If the cell is not None, the magic returns None.

        * If the cell evaluates as False, the resulting value is returned
        unless the final line prints something to the console, in
        which case None is returned.

        * If the final line results in a NULL value when evaluated
        by rpy2, then None is returned.

        * No attempt is made to convert the final value to a structured array.
        Use the --dataframe flag or %Rget to push / return a structured array.

        * If the -n flag is present, there is no return value.

        * A trailing ';' will also result in no return value as the last
        value in the line is an empty string.

        The --dataframe argument will attempt to return structured arrays. 
        This is useful for dataframes with
        mixed data types. Note also that for a data.frame, 
        if it is returned as an ndarray, it is transposed::

            In [18]: dtype=[('x', '<i4'), ('y', '<f8'), ('z', '|S1')]

            In [19]: datapy = np.array([(1, 2.9, 'a'), (2, 3.5, 'b'), (3, 2.1, 'c'), (4, 5, 'e')], dtype=dtype)

            In [20]: %%R -o datar
            datar = datapy
               ....: 

            In [21]: datar
            Out[21]: 
            array([['1', '2', '3', '4'],
                   ['2', '3', '2', '5'],
                   ['a', 'b', 'c', 'e']], 
                  dtype='|S1')

            In [22]: %%R -d datar
            datar = datapy
               ....: 

            In [23]: datar
            Out[23]: 
            array([(1, 2.9, 'a'), (2, 3.5, 'b'), (3, 2.1, 'c'), (4, 5.0, 'e')], 
                  dtype=[('x', '<i4'), ('y', '<f8'), ('z', '|S1')])

        The --dataframe argument first tries colnames, then names.
        If both are NULL, it returns an ndarray (i.e. unstructured)::

            In [1]: %R mydata=c(4,6,8.3); NULL

            In [2]: %R -d mydata

            In [3]: mydata
            Out[3]: array([ 4. ,  6. ,  8.3])

            In [4]: %R names(mydata) = c('a','b','c'); NULL

            In [5]: %R -d mydata

            In [6]: mydata
            Out[6]: 
            array((4.0, 6.0, 8.3), 
                  dtype=[('a', '<f8'), ('b', '<f8'), ('c', '<f8')])

            In [7]: %R -o mydata

            In [8]: mydata
            Out[8]: array([ 4. ,  6. ,  8.3])

        '''

        args = parse_argstring(self.R, line)

        # arguments 'code' in line are prepended to
        # the cell lines

        if cell is None:
            code = ''
            return_output = True
            line_mode = True
        else:
            code = cell
            return_output = False
            line_mode = False

        code = ' '.join(args.code) + code

        # if there is no local namespace then default to an empty dict
        if local_ns is None:
            local_ns = {}

        if args.input:
            for input in ','.join(args.input).split(','):
                try:
                    val = local_ns[input]
                except KeyError:
                    val = self.shell.user_ns[input]
                self.r.assign(input, self.pyconverter(val))

        png_argdict = dict([(n, getattr(args, n)) for n in ['units', 'height', 'width', 'bg', 'pointsize']])
        png_args = ','.join(['%s=%s' % (o,v) for o, v in list(png_argdict.items()) if v is not None])
        # execute the R code in a temporary directory

        tmpd = tempfile.mkdtemp()
        self.r('png("%s/Rplots%%03d.png",%s)' % (tmpd, png_args))

        text_output = ''
        if line_mode:
            for line in code.split(';'):
                text_result, result = self.eval(line)
                text_output += text_result
            if text_result:
                # the last line printed something to the console so we won't return it
                return_output = False
        else:
            text_result, result = self.eval(code)
            text_output += text_result

        self.r('dev.off()')

        # read out all the saved .png files

        images = [open(imgfile, 'rb').read() for imgfile in glob("%s/Rplots*png" % tmpd)]

        # now publish the images
        # mimicking IPython/zmq/pylab/backend_inline.py
        fmt = 'png'
        mimetypes = { 'png' : 'image/png', 'svg' : 'image/svg+xml' }
        mime = mimetypes[fmt]

        # publish the printed R objects, if any

        display_data = []
        if text_output:
            display_data.append(('RMagic.R', {'text/plain':text_output}))

        # flush text streams before sending figures, helps a little with output
        for image in images:
            # synchronization in the console (though it's a bandaid, not a real sln)
            sys.stdout.flush(); sys.stderr.flush()
            display_data.append(('RMagic.R', {mime: image}))

        # kill the temporary directory
        rmtree(tmpd)

        # try to turn every output into a numpy array
        # this means that output are assumed to be castable
        # as numpy arrays

        if args.output:
            for output in ','.join(args.output).split(','):
                self.shell.push({output:self.Rconverter(self.r(output), dataframe=False)})

        if args.dataframe:
            for output in ','.join(args.dataframe).split(','):
                self.shell.push({output:self.Rconverter(self.r(output), dataframe=True)})

        for tag, disp_d in display_data:
            publish_display_data(tag, disp_d)

        # this will keep a reference to the display_data
        # which might be useful to other objects who happen to use
        # this method

        if self.cache_display_data:
            self.display_cache = display_data

        # if in line mode and return_output, return the result as an ndarray
        if return_output and not args.noreturn:
            if result != ri.NULL:
                return self.Rconverter(result, dataframe=False)

Example 102

View license
def main(_):
  if FLAGS.self_test:
    print('Running self-test.')
    train_data, train_labels = fake_data(256)
    validation_data, validation_labels = fake_data(EVAL_BATCH_SIZE)
    test_data, test_labels = fake_data(EVAL_BATCH_SIZE)
    num_epochs = 1
  else:
    # Get the data.
    train_data_filename = maybe_download('train-images-idx3-ubyte.gz')
    train_labels_filename = maybe_download('train-labels-idx1-ubyte.gz')
    test_data_filename = maybe_download('t10k-images-idx3-ubyte.gz')
    test_labels_filename = maybe_download('t10k-labels-idx1-ubyte.gz')

    # Extract it into numpy arrays.
    train_data = extract_data(train_data_filename, 60000)
    train_labels = extract_labels(train_labels_filename, 60000)
    test_data = extract_data(test_data_filename, 10000)
    test_labels = extract_labels(test_labels_filename, 10000)

    # Generate a validation set.
    validation_data = train_data[:VALIDATION_SIZE, ...]
    validation_labels = train_labels[:VALIDATION_SIZE]
    train_data = train_data[VALIDATION_SIZE:, ...]
    train_labels = train_labels[VALIDATION_SIZE:]
    num_epochs = NUM_EPOCHS
  train_size = train_labels.shape[0]

  # This is where training samples and labels are fed to the graph.
  # These placeholder nodes will be fed a batch of training data at each
  # training step using the {feed_dict} argument to the Run() call below.
  train_data_node = tf.placeholder(
      data_type(),
      shape=(BATCH_SIZE, IMAGE_SIZE, IMAGE_SIZE, NUM_CHANNELS))
  train_labels_node = tf.placeholder(tf.int64, shape=(BATCH_SIZE,))
  eval_data = tf.placeholder(
      data_type(),
      shape=(EVAL_BATCH_SIZE, IMAGE_SIZE, IMAGE_SIZE, NUM_CHANNELS))

  # The variables below hold all the trainable weights. They are passed an
  # initial value which will be assigned when we call:
  # {tf.global_variables_initializer().run()}
  conv1_weights = tf.Variable(
      tf.truncated_normal([5, 5, NUM_CHANNELS, 32],  # 5x5 filter, depth 32.
                          stddev=0.1,
                          seed=SEED, dtype=data_type()))
  conv1_biases = tf.Variable(tf.zeros([32], dtype=data_type()))
  conv2_weights = tf.Variable(tf.truncated_normal(
      [5, 5, 32, 64], stddev=0.1,
      seed=SEED, dtype=data_type()))
  conv2_biases = tf.Variable(tf.constant(0.1, shape=[64], dtype=data_type()))
  fc1_weights = tf.Variable(  # fully connected, depth 512.
      tf.truncated_normal([IMAGE_SIZE // 4 * IMAGE_SIZE // 4 * 64, 512],
                          stddev=0.1,
                          seed=SEED,
                          dtype=data_type()))
  fc1_biases = tf.Variable(tf.constant(0.1, shape=[512], dtype=data_type()))
  fc2_weights = tf.Variable(tf.truncated_normal([512, NUM_LABELS],
                                                stddev=0.1,
                                                seed=SEED,
                                                dtype=data_type()))
  fc2_biases = tf.Variable(tf.constant(
      0.1, shape=[NUM_LABELS], dtype=data_type()))

  # We will replicate the model structure for the training subgraph, as well
  # as the evaluation subgraphs, while sharing the trainable parameters.
  def model(data, train=False):
    """The Model definition."""
    # 2D convolution, with 'SAME' padding (i.e. the output feature map has
    # the same size as the input). Note that {strides} is a 4D array whose
    # shape matches the data layout: [image index, y, x, depth].
    conv = tf.nn.conv2d(data,
                        conv1_weights,
                        strides=[1, 1, 1, 1],
                        padding='SAME')
    # Bias and rectified linear non-linearity.
    relu = tf.nn.relu(tf.nn.bias_add(conv, conv1_biases))
    # Max pooling. The kernel size spec {ksize} also follows the layout of
    # the data. Here we have a pooling window of 2, and a stride of 2.
    pool = tf.nn.max_pool(relu,
                          ksize=[1, 2, 2, 1],
                          strides=[1, 2, 2, 1],
                          padding='SAME')
    conv = tf.nn.conv2d(pool,
                        conv2_weights,
                        strides=[1, 1, 1, 1],
                        padding='SAME')
    relu = tf.nn.relu(tf.nn.bias_add(conv, conv2_biases))
    pool = tf.nn.max_pool(relu,
                          ksize=[1, 2, 2, 1],
                          strides=[1, 2, 2, 1],
                          padding='SAME')
    # Reshape the feature map cuboid into a 2D matrix to feed it to the
    # fully connected layers.
    pool_shape = pool.get_shape().as_list()
    reshape = tf.reshape(
        pool,
        [pool_shape[0], pool_shape[1] * pool_shape[2] * pool_shape[3]])
    # Fully connected layer. Note that the '+' operation automatically
    # broadcasts the biases.
    hidden = tf.nn.relu(tf.matmul(reshape, fc1_weights) + fc1_biases)
    # Add a 50% dropout during training only. Dropout also scales
    # activations such that no rescaling is needed at evaluation time.
    if train:
      hidden = tf.nn.dropout(hidden, 0.5, seed=SEED)
    return tf.matmul(hidden, fc2_weights) + fc2_biases

  # Training computation: logits + cross-entropy loss.
  logits = model(train_data_node, True)
  loss = tf.reduce_mean(tf.nn.sparse_softmax_cross_entropy_with_logits(
      logits, train_labels_node))

  # L2 regularization for the fully connected parameters.
  regularizers = (tf.nn.l2_loss(fc1_weights) + tf.nn.l2_loss(fc1_biases) +
                  tf.nn.l2_loss(fc2_weights) + tf.nn.l2_loss(fc2_biases))
  # Add the regularization term to the loss.
  loss += 5e-4 * regularizers

  # Optimizer: set up a variable that's incremented once per batch and
  # controls the learning rate decay.
  batch = tf.Variable(0, dtype=data_type())
  # Decay once per epoch, using an exponential schedule starting at 0.01.
  learning_rate = tf.train.exponential_decay(
      0.01,                # Base learning rate.
      batch * BATCH_SIZE,  # Current index into the dataset.
      train_size,          # Decay step.
      0.95,                # Decay rate.
      staircase=True)
  # Use simple momentum for the optimization.
  optimizer = tf.train.MomentumOptimizer(learning_rate,
                                         0.9).minimize(loss,
                                                       global_step=batch)

  # Predictions for the current training minibatch.
  train_prediction = tf.nn.softmax(logits)

  # Predictions for the test and validation, which we'll compute less often.
  eval_prediction = tf.nn.softmax(model(eval_data))

  # Small utility function to evaluate a dataset by feeding batches of data to
  # {eval_data} and pulling the results from {eval_predictions}.
  # Saves memory and enables this to run on smaller GPUs.
  def eval_in_batches(data, sess):
    """Get all predictions for a dataset by running it in small batches."""
    size = data.shape[0]
    if size < EVAL_BATCH_SIZE:
      raise ValueError("batch size for evals larger than dataset: %d" % size)
    predictions = numpy.ndarray(shape=(size, NUM_LABELS), dtype=numpy.float32)
    for begin in xrange(0, size, EVAL_BATCH_SIZE):
      end = begin + EVAL_BATCH_SIZE
      if end <= size:
        predictions[begin:end, :] = sess.run(
            eval_prediction,
            feed_dict={eval_data: data[begin:end, ...]})
      else:
        batch_predictions = sess.run(
            eval_prediction,
            feed_dict={eval_data: data[-EVAL_BATCH_SIZE:, ...]})
        predictions[begin:, :] = batch_predictions[begin - size:, :]
    return predictions

  # Create a local session to run the training.
  start_time = time.time()
  with tf.Session() as sess:
    # Run all the initializers to prepare the trainable parameters.
    tf.global_variables_initializer().run()
    print('Initialized!')
    # Loop through training steps.
    for step in xrange(int(num_epochs * train_size) // BATCH_SIZE):
      # Compute the offset of the current minibatch in the data.
      # Note that we could use better randomization across epochs.
      offset = (step * BATCH_SIZE) % (train_size - BATCH_SIZE)
      batch_data = train_data[offset:(offset + BATCH_SIZE), ...]
      batch_labels = train_labels[offset:(offset + BATCH_SIZE)]
      # This dictionary maps the batch data (as a numpy array) to the
      # node in the graph it should be fed to.
      feed_dict = {train_data_node: batch_data,
                   train_labels_node: batch_labels}
      # Run the optimizer to update weights.
      sess.run(optimizer, feed_dict=feed_dict)
      # print some extra information once reach the evaluation frequency
      if step % EVAL_FREQUENCY == 0:
        # fetch some extra nodes' data
        l, lr, predictions = sess.run([loss, learning_rate, train_prediction],
                                      feed_dict=feed_dict)
        elapsed_time = time.time() - start_time
        start_time = time.time()
        print('Step %d (epoch %.2f), %.1f ms' %
              (step, float(step) * BATCH_SIZE / train_size,
               1000 * elapsed_time / EVAL_FREQUENCY))
        print('Minibatch loss: %.3f, learning rate: %.6f' % (l, lr))
        print('Minibatch error: %.1f%%' % error_rate(predictions, batch_labels))
        print('Validation error: %.1f%%' % error_rate(
            eval_in_batches(validation_data, sess), validation_labels))
        sys.stdout.flush()
    # Finally print the result!
    test_error = error_rate(eval_in_batches(test_data, sess), test_labels)
    print('Test error: %.1f%%' % test_error)
    if FLAGS.self_test:
      print('test_error', test_error)
      assert test_error == 0.0, 'expected 0.0 test_error, got %.2f' % (
          test_error,)

Example 103

Project: metaseq
Source File: signal_comparison.py
View license
def compare(signal1, signal2, features, outfn, comparefunc=np.subtract,
        batchsize=5000, array_kwargs=None, verbose=False):
    """
    Compares two genomic signal objects and outputs results as a bedGraph file.
    Can be used for entire genome-wide comparisons due to its parallel nature.

    Typical usage would be to create genome-wide windows of equal size to
    provide as `features`::

        windowsize = 10000
        features = pybedtools.BedTool().window_maker(
           genome='hg19', w=windowsize)

    You will usually want to choose bins for the array based on the final
    resolution you would like. Say you would like 10-bp bins in the final
    bedGraph; using the example above you would use array_kwargs={'bins':
    windowsize/10}.  Or, for single-bp resolution (beware: file will be large),
    use {'bins': windowsize}.

    Here's how it works.  This function:

        * Takes `batchsize` features at a time from `features`

        * Constructs normalized (RPMMR) arrays in parallel for each input
          genomic signal object for those `batchsize` features

        * Applies `comparefunc` (np.subtract by default) to the arrays to get
          a "compared" (e.g., difference matrix by default) for the `batchsize`
          features.

        * For each row in this matrix, it outputs each nonzero column as
          a bedGraph format line in `outfn`

    `comparefunc` is a function with the signature::

        def f(x, y):
            return z

    where `x` and `y` will be arrays for `signal1` and `signal2` (normalized to
    RPMMR) and `z` is a new array.  By default this is np.subtract, but another
    common `comparefunc` might be a log2-fold-change function::

        def lfc(x, y):
            return np.log2(x / y)

    :param signal1: A genomic_signal object
    :param signal2: Another genomic_signal object
    :param features: An iterable of pybedtools.Interval objects. A list will be
        created for every `batchsize` features, so you need enough memory for
        this.
    :param comparefunc: Function to use to compare arrays (default is
        np.subtract)
    :param outfn: String filename to write bedGraph file
    :param batchsize: Number of features (each with length `windowsize` bp) to
        process at a time
    :param array_kwargs: Kwargs passed directly to genomic_signal.array.  Needs
        `processes` and `chunksize` if you want parallel processing
    :param verbose: Be noisy
    """
    fout = open(outfn, 'w')
    fout.write('track type=bedGraph\n')

    i = 0
    this_batch = []
    for feature in features:
        if i <= batchsize:
            this_batch.append(feature)
            i += 1
            continue

        if verbose:
            print 'working on batch of %s' % batchsize
            sys.stdout.flush()

        arr1 = signal1.array(this_batch, **array_kwargs).astype(float)
        arr2 = signal2.array(this_batch, **array_kwargs).astype(float)
        arr1 /= signal1.million_mapped_reads()
        arr2 /= signal2.million_mapped_reads()
        compared = comparefunc(arr1, arr2)

        for feature, row in itertools.izip(this_batch, compared):
            start = feature.start
            bins = len(row)
            binsize = len(feature) / len(row)

            # Quickly move on if nothing here.  speed increase prob best for
            # sparse data
            if sum(row) == 0:
                continue

            for j in range(0, len(row)):
                score = row[j]
                stop = start + binsize
                if score != 0:
                    fout.write('\t'.join([
                        feature.chrom,
                        str(start),
                        str(stop),
                        str(score)]) + '\n')
                start = start + binsize
        this_batch = []
        i = 0
    fout.close()

Example 104

Project: stoq
Source File: logo.py
View license
def print_logo():
    """

    print random ascii art

    """

    logo = []

    logo.append("""
    .------..------..------..------.
    |S.--. ||T.--. ||O.--. ||Q.--. |
    | :/\: || :/\: || :/\: || (\/) |
    | :\/: || (__) || :\/: || :\/: |
    | '--'S|| '--'T|| '--'O|| '--'Q|
    `------'`------'`------'`------'

          Analysis. Simplified.
                 v{}
    """.format(__version__))

    logo.append("""
          *******                               * ***
        *       ***      *                    *  ****
       *         **     **                   *  *  ***
       **        *      **                  *  **   ***
        ***           ********    ****     *  ***    ***
       ** ***        ********    * ***  * **   **     **
        *** ***         **      *   ****  **   **     **
          *** ***       **     **    **   **   **     **
            *** ***     **     **    **   **   **     **
              ** ***    **     **    **   **   **     **
               ** **    **     **    **    **  ** *** **
                * *     **     **    **     ** *   ****
      ***        *      **      ******       ***     ***
     *  *********        **      ****         ******* **
    *     *****                                 ***   **
    *                                                 **
     **                                               *
                                                     *
                                                    *
                    Analysis. Simplified.
                          v{}
    """.format(__version__))

    logo.append("""
     .d8888b.  888             .d88888b.
    d88P  Y88b 888            d88P" "Y88b
    Y88b.      888            888     888
     "Y888b.   888888 .d88b.  888     888
        "Y88b. 888   d88""88b 888     888
          "888 888   888  888 888 Y8b 888
    Y88b  d88P Y88b. Y88..88P Y88b.Y8b88P
     "Y8888P"   "Y888 "Y88P"   "Y888888"
                                     Y8b
            Analysis. Simplified.
                  v{}
    """.format(__version__))

    logo.append("""
     _______ _______  _____   _____
     |______    |    |     | |   __|
     ______|    |    |_____| |____\|

           Analysis. Simplified.
                 v{}
    """.format(__version__))

    logo.append("""
      .--.--.       ___                  ,----..
     /  /    '.   ,--.'|_               /   /   '
    |  :  /`. /   |  | :,'    ,---.    /   .     :
    ;  |  |--`    :  : ' :   '   ,'\  .   /   ;.  '
    |  :  ;_    .;__,'  /   /   /   |.   ;   /  ` ;
     \  \    `. |  |   |   .   ; ,. :;   |  ; \ ; |
      `----.   \:__,'| :   '   | |: :|   :  | ; | '
      __ \  \  |  '  : |__ '   | .; :.   |  ' ' ' :
     /  /`--'  /  |  | '.'||   :    |'   ;  \; /  |
    '--'.     /   ;  :    ; \   \  /  \   \  ',  . "
      `--'---'    |  ,   /   `----'    ;   :      ; |
                   ---`-'               \   \ .'`--"
                                         `---`
                Analysis. Simplified.
                      v{}
    """.format(__version__))

    logo.append("""
     _______ _________ _______  _______
    (  ____ \\__   __/(  ___  )(  ___  )
    | (    \/   ) (   | (   ) || (   ) |
    | (_____    | |   | |   | || |   | |
    (_____  )   | |   | |   | || |   | |
          ) |   | |   | |   | || | /\| |
    /\____) |   | |   | (___) || (_\ \ |
    \_______)   )_(   (_______)(____\/_)

            Analysis. Simplified.
                  v{}
    """.format(__version__))

    logo.append("""
      _________  __          ________
     /   _____/_/  |_  ____  \_____  -
     \_____  \ \   __\/  _ \  /  / \  -
     /        \ |  | (  <_> )/   \_/.  -
    /_______  / |__|  \____/ \_____\ \_/
            \/                      \__>

            Analysis. Simplified.
                  v{}
    """.format(__version__))

    logo.append("""
               ___
              (   )
        .--.   | |_      .--.    .--.
      /  _  \ (   __)   /    \  /    "
     . .' `. ; | |     |  .-. ;|  .-. '
     | '   | | | | ___ | |  | || |  | |
     _\_`.(___)| |(   )| |  | || |  | |
    (   ). '.  | | | | | |  | || |  | |
     | |  `\ | | ' | | | '  | || '  | |
     ; '._,' ' ' `-' ; '  `-' /' `-'  |
      '.___.'   `.__.   `.__.'  `._ / |
                                    | |
                                   (___)

            Analysis. Simplified.
                  v{}
    """.format(__version__))

    logo.append("""
    ███████╗████████╗ ██████╗  ██████╗
    ██╔════╝╚══██╔══╝██╔═══██╗██╔═══██╗
    ███████╗   ██║   ██║   ██║██║   ██║
    ╚════██║   ██║   ██║   ██║██║▄▄ ██║
    ███████║   ██║   ╚██████╔╝╚██████╔╝
    ╚══════╝   ╚═╝    ╚═════╝  ╚══▀▀═╝

           Analysis. Simplified.
                 v{}
    """.format(__version__))

    logo.append("""
     ▄████████     ███      ▄██████▄  ████████▄
    ███    ███ ▀█████████▄ ███    ███ ███    ███
    ███    █▀     ▀███▀▀██ ███    ███ ███    ███
    ███            ███   ▀ ███    ███ ███    ███
    ▀███████████   ███     ███    ███ ███    ███
             ███   ███     ███    ███ ███    ███
       ▄█    ███   ███     ███    ███ ███  ▀ ███
     ▄████████▀   ▄████▀    ▀██████▀   ▀██████▀▄█

                   Analysis. Simplified.
                         v{}
    """.format(__version__))

    sys.stdout.flush()
    try:
        print(random.choice(logo))
    except:
        print(logo[3])
    sys.stdout.flush()

Example 105

Project: PythonJS
Source File: run.py
View license
def run_test_on(filename):
    """run one test and returns the number of errors"""
    if not show_details:
        f = open(filename)
        comment = f.readline().strip(" \n\"'")
        f.close()
        print(table_header % (filename[2:-3], comment), end='')
    sum_errors = {}

    if filename.endswith('.html'):
        run_html_test( filename, sum_errors )
        return sum_errors

    def display(function):
        global _test_description
        _test_description = function.__doc__
        if show_details:
            print('\n<%s>\n' % function.__doc__)

        errors = function(filename)
        if errors:
            if not show_details:
                print(table_cell % ''.join('%s%d' % (k[0], v)
                                            for k, v in errors.items()),
                      end='')
        else:
            if not show_details:
                print(table_cell % 'OK', end='')
        sys.stdout.flush()

        for k, v in errors.items():
            sum_errors[k] = sum_errors.get(k, 0) + v

        if show_details:
            print('-'*77)

    if 'requirejs' not in filename and not filename.startswith('./go/'):
        display(run_python_test_on)
        display(run_python3_test_on)
        if pypy_runnable:
            display(run_pypy_test_on)
        if old_pypy_runnable:
            display(run_old_pypy_test_on)

    global js
    if not filename.startswith('./go/'):
        js = translate_js(
            filename, 
            javascript=False, 
            multioutput=filename.startswith('./threads/' or filename.startswith('./bench/webworker'))
        )
        if rhino_runnable:
            display(run_pythonjs_test_on)
        if node_runnable:
            display(run_pythonjs_test_on_node)

        if nodewebkit_runnable:
            display(run_pythonjs_test_on_nodewebkit)


    if '--no-javascript-mode' not in sys.argv and not filename.startswith('./go/'):
        js = translate_js(filename, javascript=True, multioutput=filename.startswith('./threads/' or filename.startswith('./bench/webworker')))
        if rhino_runnable:
            display(run_pythonjsjs_test_on)
        if node_runnable:
            display(run_pythonjsjs_test_on_node)

        if nodewebkit_runnable:
            display(run_pythonjsjs_test_on_nodewebkit)


    if 'requirejs' not in filename:

        if dart_runnable:
            js = translate_js(filename, javascript=False, dart=True)
            display(run_pythonjs_dart_test_on_dart)

        if dart2js_runnable and node_runnable:
            js = translate_js(filename, javascript=False, dart=True)
            display(run_pythonjs_dart_test_on_node)

        if coffee_runnable and node_runnable:
            js = translate_js(filename, javascript=False, dart=False, coffee=True)
            display(run_pythonjs_coffee_test_on_node)

        if luajs_runnable and node_runnable:
            js = translate_js(filename, luajs=True)
            display(run_pythonjs_luajs_test_on_node)

        if lua_runnable:
            js = translate_js(filename, lua=True)
            display(run_pythonjs_lua_test_on_lua)

        if luajit_runnable:
            js = translate_js(filename, lua=True)
            display(run_pythonjs_lua_test_on_luajit)

        if go_runnable:
            js = translate_js(filename, go=True)
            display(run_pythonjs_go_test)

        if gopherjs_runnable:
            js = translate_js(filename, gopherjs=True)
            display(run_pythonjs_gopherjs_test)

    print()
    return sum_errors

Example 106

Project: apilogs
Source File: core.py
View license
    def list_logs(self):
        streams = []

        if self.log_stream_name != self.ALL_WILDCARD:
            streams = list(self._get_streams_from_pattern(self.log_group_name, self.log_stream_name))

            if len(streams) > self.FILTER_LOG_EVENTS_STREAMS_LIMIT:
                raise exceptions.TooManyStreamsFilteredError(
                     self.log_stream_name,
                     len(streams),
                     self.FILTER_LOG_EVENTS_STREAMS_LIMIT
                )
            if len(streams) == 0:
                raise exceptions.NoStreamsFilteredError(self.log_stream_name)

        max_stream_length = max([len(s) for s in streams]) if streams else 10
        group_length = len(self.log_group_name)

        queue, exit = Queue(), Event()

        def update_next_token(response, kwargs):
            group = kwargs['logGroupName']

            if 'nextToken' in response:
                next = response['nextToken']
    
                self.next_tokens[group] = next

                #print "Updated tokens"
                #print self.next_tokens
            else:
                if group in self.next_tokens:
                    del self.next_tokens[group]

                if self.watch:
                    time.sleep(0.2)
                else:
                    queue.put(None)
                    return

        ## todo: remove shared kwargs
        def list_lambda_logs(allevents, kwargs):
            # add events from lambda function streams
            fxns = self.get_lambda_function_names(self.api_id, self.stage)
            for fxn in fxns:
                lambda_group = ("/aws/lambda/" + fxn).split(':')[0]
                kwargs['logGroupName'] = lambda_group
    
                if lambda_group in self.next_tokens:
                    kwargs['nextToken'] = self.next_tokens[lambda_group]
                else:
                    if 'nextToken' in kwargs:
                        del kwargs['nextToken']

                lambdaresponse = filter_log_events(**kwargs)
                events = lambdaresponse.get('events', [])
                for event in events:
                    event['group_name'] = lambda_group
                    allevents.append(event)
                update_next_token(lambdaresponse, kwargs)
                return allevents

        ## todo: remove shared kwargs
        def list_apigateway_logs(allevents, kwargs):
            # add events from API Gateway streams
            kwargs['logGroupName'] = self.log_group_name
            if self.log_group_name in self.next_tokens:
                kwargs['nextToken'] = self.next_tokens[self.log_group_name]
            else:
                if 'nextToken' in kwargs:
                    del kwargs['nextToken']

            try:
                apigresponse = filter_log_events(**kwargs)
            except Exception as e:
                print "Error fetching logs for API {0}. Please ensure logging is enabled for this API and the API is deployed. " \
                      "See http://docs.aws.amazon.com/apigateway/latest/developerguide/how-to-stage-settings.html: {1}".format(self.api_id, e)
                raise

            events = apigresponse.get('events', [])
            for event in events:
                event['group_name'] = self.log_group_name
                allevents.append(event)
            update_next_token(apigresponse, kwargs)
            return allevents

        def filter_log_events(**kwargs):
            try:
                resp = self.client.filter_log_events(**kwargs)

                if 'nextToken' in resp:
                    group = kwargs['logGroupName']
                    next = resp['nextToken']
                    #print "Resp: Group: " + group + " nextToken: " + next

                #print resp

                return resp
            except Exception as e:
                print "Caught error from CloudWatch: {0}".format(e)
                raise


        def consumer():
            while not exit.is_set():
                event = queue.get()

                if event is None:
                    exit.set()
                    break

                output = []
                if self.output_group_enabled:
                    output.append(
                        self.color(
                            event['group_name'].ljust(group_length, ' '),
                            'green'
                        )
                    )
                if self.output_stream_enabled:
                    output.append(
                        self.color(
                            event['logStreamName'].ljust(max_stream_length,
                                                         ' '),
                            'cyan'
                        )
                    )
                if self.output_timestamp_enabled:
                    output.append(
                        self.color(
                            milis2iso(event['timestamp']),
                            'yellow'
                        )
                    )
                if self.output_ingestion_time_enabled:
                    output.append(
                        self.color(
                            milis2iso(event['ingestionTime']),
                            'blue'
                        )
                    )
                output.append(event['message'])
                print(' '.join(output))
                sys.stdout.flush()

        def generator():
            """Push events into queue trying to deduplicate them using a lru queue.
            AWS API stands for the interleaved parameter that:
                interleaved (boolean) -- If provided, the API will make a best
                effort to provide responses that contain events from multiple
                log streams within the log group interleaved in a single
                response. That makes some responses return some subsequent
                response duplicate events. In a similar way when awslogs is
                called with --watch option, we need to findout which events we
                have alredy put in the queue in order to not do it several
                times while waiting for new ones and reusing the same
                next_token. The site of this queue is MAX_EVENTS_PER_CALL in
                order to not exhaust the memory.
            """
            interleaving_sanity = deque(maxlen=self.MAX_EVENTS_PER_CALL)
            kwargs = {'logGroupName': self.log_group_name,
                      'interleaved': True}

            if streams:
                kwargs['logStreamNames'] = streams

            if self.start:
                kwargs['startTime'] = self.start

            if self.end:
                kwargs['endTime'] = self.end

            if self.filter_pattern:
                kwargs['filterPattern'] = self.filter_pattern

            while not exit.is_set():
                allevents = []
                
                list_apigateway_logs(allevents, kwargs)
                list_lambda_logs(allevents, kwargs)

                sorted(allevents, key=itemgetter('timestamp'))

                for event in allevents:
                    if event['eventId'] not in interleaving_sanity:
                        interleaving_sanity.append(event['eventId'])
                        queue.put(event)

                #print response

        g = Thread(target=generator)
        g.start()

        c = Thread(target=consumer)
        c.start()

        try:
            while not exit.is_set():
                time.sleep(.1)
        except (KeyboardInterrupt, SystemExit):
            exit.set()
            print('Closing...\n')
            os._exit(0)

Example 107

View license
def main(_):
  if FLAGS.self_test:
    print('Running self-test.')
    train_data, train_labels = fake_data(256)
    validation_data, validation_labels = fake_data(EVAL_BATCH_SIZE)
    test_data, test_labels = fake_data(EVAL_BATCH_SIZE)
    num_epochs = 1
  else:
    # Get the data.
    train_data_filename = maybe_download('train-images-idx3-ubyte.gz')
    train_labels_filename = maybe_download('train-labels-idx1-ubyte.gz')
    test_data_filename = maybe_download('t10k-images-idx3-ubyte.gz')
    test_labels_filename = maybe_download('t10k-labels-idx1-ubyte.gz')

    # Extract it into numpy arrays.
    train_data = extract_data(train_data_filename, 60000)
    train_labels = extract_labels(train_labels_filename, 60000)
    test_data = extract_data(test_data_filename, 10000)
    test_labels = extract_labels(test_labels_filename, 10000)

    # Generate a validation set.
    validation_data = train_data[:VALIDATION_SIZE, ...]
    validation_labels = train_labels[:VALIDATION_SIZE]
    train_data = train_data[VALIDATION_SIZE:, ...]
    train_labels = train_labels[VALIDATION_SIZE:]
    num_epochs = NUM_EPOCHS
  train_size = train_labels.shape[0]

  # This is where training samples and labels are fed to the graph.
  # These placeholder nodes will be fed a batch of training data at each
  # training step using the {feed_dict} argument to the Run() call below.
  train_data_node = tf.placeholder(
      data_type(),
      shape=(BATCH_SIZE, IMAGE_SIZE, IMAGE_SIZE, NUM_CHANNELS))
  train_labels_node = tf.placeholder(tf.int64, shape=(BATCH_SIZE,))
  eval_data = tf.placeholder(
      data_type(),
      shape=(EVAL_BATCH_SIZE, IMAGE_SIZE, IMAGE_SIZE, NUM_CHANNELS))

  # The variables below hold all the trainable weights. They are passed an
  # initial value which will be assigned when we call:
  # {tf.global_variables_initializer().run()}
  conv1_weights = tf.Variable(
      tf.truncated_normal([5, 5, NUM_CHANNELS, 32],  # 5x5 filter, depth 32.
                          stddev=0.1,
                          seed=SEED, dtype=data_type()))
  conv1_biases = tf.Variable(tf.zeros([32], dtype=data_type()))
  conv2_weights = tf.Variable(tf.truncated_normal(
      [5, 5, 32, 64], stddev=0.1,
      seed=SEED, dtype=data_type()))
  conv2_biases = tf.Variable(tf.constant(0.1, shape=[64], dtype=data_type()))
  fc1_weights = tf.Variable(  # fully connected, depth 512.
      tf.truncated_normal([IMAGE_SIZE // 4 * IMAGE_SIZE // 4 * 64, 512],
                          stddev=0.1,
                          seed=SEED,
                          dtype=data_type()))
  fc1_biases = tf.Variable(tf.constant(0.1, shape=[512], dtype=data_type()))
  fc2_weights = tf.Variable(tf.truncated_normal([512, NUM_LABELS],
                                                stddev=0.1,
                                                seed=SEED,
                                                dtype=data_type()))
  fc2_biases = tf.Variable(tf.constant(
      0.1, shape=[NUM_LABELS], dtype=data_type()))

  # We will replicate the model structure for the training subgraph, as well
  # as the evaluation subgraphs, while sharing the trainable parameters.
  def model(data, train=False):
    """The Model definition."""
    # 2D convolution, with 'SAME' padding (i.e. the output feature map has
    # the same size as the input). Note that {strides} is a 4D array whose
    # shape matches the data layout: [image index, y, x, depth].
    conv = tf.nn.conv2d(data,
                        conv1_weights,
                        strides=[1, 1, 1, 1],
                        padding='SAME')
    # Bias and rectified linear non-linearity.
    relu = tf.nn.relu(tf.nn.bias_add(conv, conv1_biases))
    # Max pooling. The kernel size spec {ksize} also follows the layout of
    # the data. Here we have a pooling window of 2, and a stride of 2.
    pool = tf.nn.max_pool(relu,
                          ksize=[1, 2, 2, 1],
                          strides=[1, 2, 2, 1],
                          padding='SAME')
    conv = tf.nn.conv2d(pool,
                        conv2_weights,
                        strides=[1, 1, 1, 1],
                        padding='SAME')
    relu = tf.nn.relu(tf.nn.bias_add(conv, conv2_biases))
    pool = tf.nn.max_pool(relu,
                          ksize=[1, 2, 2, 1],
                          strides=[1, 2, 2, 1],
                          padding='SAME')
    # Reshape the feature map cuboid into a 2D matrix to feed it to the
    # fully connected layers.
    pool_shape = pool.get_shape().as_list()
    reshape = tf.reshape(
        pool,
        [pool_shape[0], pool_shape[1] * pool_shape[2] * pool_shape[3]])
    # Fully connected layer. Note that the '+' operation automatically
    # broadcasts the biases.
    hidden = tf.nn.relu(tf.matmul(reshape, fc1_weights) + fc1_biases)
    # Add a 50% dropout during training only. Dropout also scales
    # activations such that no rescaling is needed at evaluation time.
    if train:
      hidden = tf.nn.dropout(hidden, 0.5, seed=SEED)
    return tf.matmul(hidden, fc2_weights) + fc2_biases

  # Training computation: logits + cross-entropy loss.
  logits = model(train_data_node, True)
  loss = tf.reduce_mean(tf.nn.sparse_softmax_cross_entropy_with_logits(
      logits, train_labels_node))

  # L2 regularization for the fully connected parameters.
  regularizers = (tf.nn.l2_loss(fc1_weights) + tf.nn.l2_loss(fc1_biases) +
                  tf.nn.l2_loss(fc2_weights) + tf.nn.l2_loss(fc2_biases))
  # Add the regularization term to the loss.
  loss += 5e-4 * regularizers

  # Optimizer: set up a variable that's incremented once per batch and
  # controls the learning rate decay.
  batch = tf.Variable(0, dtype=data_type())
  # Decay once per epoch, using an exponential schedule starting at 0.01.
  learning_rate = tf.train.exponential_decay(
      0.01,                # Base learning rate.
      batch * BATCH_SIZE,  # Current index into the dataset.
      train_size,          # Decay step.
      0.95,                # Decay rate.
      staircase=True)
  # Use simple momentum for the optimization.
  optimizer = tf.train.MomentumOptimizer(learning_rate,
                                         0.9).minimize(loss,
                                                       global_step=batch)

  # Predictions for the current training minibatch.
  train_prediction = tf.nn.softmax(logits)

  # Predictions for the test and validation, which we'll compute less often.
  eval_prediction = tf.nn.softmax(model(eval_data))

  # Small utility function to evaluate a dataset by feeding batches of data to
  # {eval_data} and pulling the results from {eval_predictions}.
  # Saves memory and enables this to run on smaller GPUs.
  def eval_in_batches(data, sess):
    """Get all predictions for a dataset by running it in small batches."""
    size = data.shape[0]
    if size < EVAL_BATCH_SIZE:
      raise ValueError("batch size for evals larger than dataset: %d" % size)
    predictions = numpy.ndarray(shape=(size, NUM_LABELS), dtype=numpy.float32)
    for begin in xrange(0, size, EVAL_BATCH_SIZE):
      end = begin + EVAL_BATCH_SIZE
      if end <= size:
        predictions[begin:end, :] = sess.run(
            eval_prediction,
            feed_dict={eval_data: data[begin:end, ...]})
      else:
        batch_predictions = sess.run(
            eval_prediction,
            feed_dict={eval_data: data[-EVAL_BATCH_SIZE:, ...]})
        predictions[begin:, :] = batch_predictions[begin - size:, :]
    return predictions

  # Create a local session to run the training.
  start_time = time.time()
  with tf.Session() as sess:
    # Run all the initializers to prepare the trainable parameters.
    tf.global_variables_initializer().run()
    print('Initialized!')
    # Loop through training steps.
    for step in xrange(int(num_epochs * train_size) // BATCH_SIZE):
      # Compute the offset of the current minibatch in the data.
      # Note that we could use better randomization across epochs.
      offset = (step * BATCH_SIZE) % (train_size - BATCH_SIZE)
      batch_data = train_data[offset:(offset + BATCH_SIZE), ...]
      batch_labels = train_labels[offset:(offset + BATCH_SIZE)]
      # This dictionary maps the batch data (as a numpy array) to the
      # node in the graph it should be fed to.
      feed_dict = {train_data_node: batch_data,
                   train_labels_node: batch_labels}
      # Run the optimizer to update weights.
      sess.run(optimizer, feed_dict=feed_dict)
      # print some extra information once reach the evaluation frequency
      if step % EVAL_FREQUENCY == 0:
        # fetch some extra nodes' data
        l, lr, predictions = sess.run([loss, learning_rate, train_prediction],
                                      feed_dict=feed_dict)
        elapsed_time = time.time() - start_time
        start_time = time.time()
        print('Step %d (epoch %.2f), %.1f ms' %
              (step, float(step) * BATCH_SIZE / train_size,
               1000 * elapsed_time / EVAL_FREQUENCY))
        print('Minibatch loss: %.3f, learning rate: %.6f' % (l, lr))
        print('Minibatch error: %.1f%%' % error_rate(predictions, batch_labels))
        print('Validation error: %.1f%%' % error_rate(
            eval_in_batches(validation_data, sess), validation_labels))
        sys.stdout.flush()
    # Finally print the result!
    test_error = error_rate(eval_in_batches(test_data, sess), test_labels)
    print('Test error: %.1f%%' % test_error)
    if FLAGS.self_test:
      print('test_error', test_error)
      assert test_error == 0.0, 'expected 0.0 test_error, got %.2f' % (
          test_error,)

Example 108

Project: CuckooSploit
Source File: logo.py
View license
def logo():
    """Cuckoo asciiarts.
    @return: asciiarts array.
    """
    logos = []

    logos.append("""
                                 _|                            
     _|_|_|  _|    _|    _|_|_|  _|  _|      _|_|      _|_|    
   _|        _|    _|  _|        _|_|      _|    _|  _|    _|  
   _|        _|    _|  _|        _|  _|    _|    _|  _|    _|  
     _|_|_|    _|_|_|    _|_|_|  _|    _|    _|_|      _|_|""")

    logos.append("""
                      __                  
  .----..--.--..----.|  |--..-----..-----.
  |  __||  |  ||  __||    < |  _  ||  _  |
  |____||_____||____||__|__||_____||_____|""")

    logos.append("""
                          .:                 
                          ::                 
    .-.     ,  :   .-.    ;;.-.  .-.   .-.   
   ;       ;   ;  ;       ;; .' ;   ;';   ;' 
   `;;;;'.'`..:;._`;;;;'_.'`  `.`;;'  `;;'""")

    logos.append("""
  eeee e   e eeee e   e  eeeee eeeee 
  8  8 8   8 8  8 8   8  8  88 8  88 
  8e   8e  8 8e   8eee8e 8   8 8   8 
  88   88  8 88   88   8 8   8 8   8 
  88e8 88ee8 88e8 88   8 8eee8 8eee8""")

    logos.append("""
  _____________________________________/\/\_______________________________
  ___/\/\/\/\__/\/\__/\/\____/\/\/\/\__/\/\__/\/\____/\/\/\______/\/\/\___
  _/\/\________/\/\__/\/\__/\/\________/\/\/\/\____/\/\__/\/\__/\/\__/\/\_
  _/\/\________/\/\__/\/\__/\/\________/\/\/\/\____/\/\__/\/\__/\/\__/\/\_
  ___/\/\/\/\____/\/\/\/\____/\/\/\/\__/\/\__/\/\____/\/\/\______/\/\/\___
  ________________________________________________________________________""")

    logos.append("""
   _______ _     _ _______ _     _  _____   _____ 
   |       |     | |       |____/  |     | |     |
   |_____  |_____| |_____  |    \\_ |_____| |_____|""")

    logos.append("""
                     _ 
    ____ _   _  ____| |  _ ___   ___
   / ___) | | |/ ___) |_/ ) _ \ / _ \\
  ( (___| |_| ( (___|  _ ( |_| | |_| |
   \\____)____/ \\____)_| \\_)___/ \\___/""")

    logos.append("""
   ______   __  __   ______   ___   ___   ______   ______      
  /_____/\\ /_/\\/_/\\ /_____/\\ /___/\\/__/\\ /_____/\\ /_____/\\     
  \\:::__\\/ \\:\\ \\:\\ \\\\:::__\\/ \\::.\\ \\\\ \\ \\\\:::_ \\ \\\\:::_ \\ \\    
   \\:\\ \\  __\\:\\ \\:\\ \\\\:\\ \\  __\\:: \\/_) \\ \\\\:\\ \\ \\ \\\\:\\ \\ \\ \\   
    \\:\\ \\/_/\\\\:\\ \\:\\ \\\\:\\ \\/_/\\\\:. __  ( ( \\:\\ \\ \\ \\\\:\\ \\ \\ \\  
     \\:\\_\\ \\ \\\\:\\_\\:\\ \\\\:\\_\\ \\ \\\\: \\ )  \\ \\ \\:\\_\\ \\ \\\\:\\_\\ \\ \\ 
      \\_____\\/ \\_____\\/ \\_____\\/ \\__\\/\\__\\/  \\_____\\/ \\_____\\/""")

    logos.append("""
    sSSs   .S       S.     sSSs   .S    S.     sSSs_sSSs      sSSs_sSSs    
   d%%SP  .SS       SS.   d%%SP  .SS    SS.   d%%SP~YS%%b    d%%SP~YS%%b   
  d%S'    S%S       S%S  d%S'    S%S    S&S  d%S'     `S%b  d%S'     `S%b  
  S%S     S%S       S%S  S%S     S%S    d*S  S%S       S%S  S%S       S%S  
  S&S     S&S       S&S  S&S     S&S   .S*S  S&S       S&S  S&S       S&S  
  S&S     S&S       S&S  S&S     S&S_sdSSS   S&S       S&S  S&S       S&S  
  S&S     S&S       S&S  S&S     S&S~YSSY%b  S&S       S&S  S&S       S&S  
  S&S     S&S       S&S  S&S     S&S    `S%  S&S       S&S  S&S       S&S  
  S*b     S*b       d*S  S*b     S*S     S%  S*b       d*S  S*b       d*S  
  S*S.    S*S.     .S*S  S*S.    S*S     S&  S*S.     .S*S  S*S.     .S*S  
   SSSbs   SSSbs_sdSSS    SSSbs  S*S     S&   SSSbs_sdSSS    SSSbs_sdSSS   
    YSSP    YSSP~YSSY      YSSP  S*S     SS    YSSP~YSSY      YSSP~YSSY    
                                 SP                                        
                                 Y""")

    logos.append("""
           _______                   _____                    _____          
          /::\\    \\                 /\\    \\                  /\\    \\         
         /::::\\    \\               /::\\____\\                /::\\    \\        
        /::::::\\    \\             /::::|   |               /::::\\    \\       
       /::::::::\\    \\           /:::::|   |              /::::::\\    \\      
      /:::/~~\\:::\\    \\         /::::::|   |             /:::/\\:::\\    \\     
     /:::/    \\:::\\    \\       /:::/|::|   |            /:::/  \\:::\\    \\    
    /:::/    / \\:::\\    \\     /:::/ |::|   |           /:::/    \\:::\\    \\   
   /:::/____/   \\:::\\____\\   /:::/  |::|___|______    /:::/    / \\:::\\    \\  
  |:::|    |     |:::|    | /:::/   |::::::::\\    \\  /:::/    /   \\:::\\ ___\\ 
  |:::|____|     |:::|    |/:::/    |:::::::::\\____\\/:::/____/  ___\\:::|    |
   \\:::\\    \\   /:::/    / \\::/    / ~~~~~/:::/    /\\:::\\    \\ /\\  /:::|____|
    \\:::\\    \\ /:::/    /   \\/____/      /:::/    /  \\:::\\    /::\\ \\::/    / 
     \\:::\\    /:::/    /                /:::/    /    \\:::\\   \\:::\\ \\/____/  
      \\:::\\__/:::/    /                /:::/    /      \\:::\\   \\:::\\____\\    
       \\::::::::/    /                /:::/    /        \\:::\\  /:::/    /    
        \\::::::/    /                /:::/    /          \\:::\\/:::/    /     
         \\::::/    /                /:::/    /            \\::::::/    /      
          \\::/____/                /:::/    /              \\::::/    /       
           ~~                      \\::/    /                \\::/____/        
                                    \\/____/                                  
                                                       it's Cuckoo!""")

    logos.append("""
            _       _                   _             _              _            _       
          /\\ \\     /\\_\\               /\\ \\           /\\_\\           /\\ \\         /\\ \\     
         /  \\ \\   / / /         _    /  \\ \\         / / /  _       /  \\ \\       /  \\ \\    
        / /\\ \\ \\  \\ \\ \\__      /\\_\\ / /\\ \\ \\       / / /  /\\_\\    / /\\ \\ \\     / /\\ \\ \\   
       / / /\\ \\ \\  \\ \\___\\    / / // / /\\ \\ \\     / / /__/ / /   / / /\\ \\ \\   / / /\\ \\ \\  
      / / /  \\ \\_\\  \\__  /   / / // / /  \\ \\_\\   / /\\_____/ /   / / /  \\ \\_\\ / / /  \\ \\_\\ 
     / / /    \\/_/  / / /   / / // / /    \\/_/  / /\\_______/   / / /   / / // / /   / / / 
    / / /          / / /   / / // / /          / / /\\ \\ \\     / / /   / / // / /   / / /  
   / / /________  / / /___/ / // / /________  / / /  \\ \\ \\   / / /___/ / // / /___/ / /   
  / / /_________\\/ / /____\\/ // / /_________\\/ / /    \\ \\ \\ / / /____\\/ // / /____\\/ /    
  \\/____________/\\/_________/ \\/____________/\\/_/      \\_\\_\\\\/_________/ \\/_________/""")

    logos.append("""
                               ),-.     /
  Cuckoo Sandbox              <(a  `---',' 
     no chance for malwares!  ( `-, ._> )
                               ) _>.___/
                                   _/""")

    logos.append("""
  .-----------------.
  | Cuckoo Sandbox? |
  |     OH NOES!    |\\  '-.__.-'   
  '-----------------' \\  /oo |--.--,--,--.
                         \\_.-'._i__i__i_.'
                               \"\"\"\"\"\"\"\"\"""")

    print(color(random.choice(logos), random.randrange(31, 37)))
    print
    print(" Cuckoo Sandbox %s" % yellow(CUCKOO_VERSION))
    print(" www.cuckoosandbox.org")
    print(" Copyright (c) 2010-2015")
    print
    sys.stdout.flush()

Example 109

Project: tensorflow-shakespeare
Source File: translate.py
View license
def train():
  """Train a en->fr translation model using WMT data."""
  # Prepare WMT data.
  # print("Preparing WMT data in %s" % FLAGS.data_dir)
  # en_train, fr_train, en_dev, fr_dev, _, _ = data_utils.prepare_wmt_data(
      # FLAGS.data_dir, FLAGS.en_vocab_size, FLAGS.fr_vocab_size)

  en_train = FLAGS.en_train
  fr_train = FLAGS.fr_train
  en_dev = FLAGS.en_dev
  fr_dev = FLAGS.fr_dev

  print("en_train", en_train)
  print("fr_train", fr_train)
  print("en_dev", en_dev)
  print("fr_dev", fr_dev)

  with tf.Session() as sess:
    # Create model.
    print("Creating %d layers of %d units." % (FLAGS.num_layers, FLAGS.size))
    model = create_model(sess, False)

    # Read data into buckets and compute their sizes.
    print ("Reading development and training data (limit: %d)."
           % FLAGS.max_train_data_size)
    dev_set = read_data(en_dev, fr_dev)
    train_set = read_data(en_train, fr_train, FLAGS.max_train_data_size)
    train_bucket_sizes = [len(train_set[b]) for b in xrange(len(_buckets))]
    train_total_size = float(sum(train_bucket_sizes))

    # A bucket scale is a list of increasing numbers from 0 to 1 that we'll use
    # to select a bucket. Length of [scale[i], scale[i+1]] is proportional to
    # the size if i-th training bucket, as used later.
    train_buckets_scale = [sum(train_bucket_sizes[:i + 1]) / train_total_size
                           for i in xrange(len(train_bucket_sizes))]

    # This is the training loop.
    step_time, loss = 0.0, 0.0
    current_step = 0
    previous_losses = []
    while True:
      # Choose a bucket according to data distribution. We pick a random number
      # in [0, 1] and use the corresponding interval in train_buckets_scale.
      random_number_01 = np.random.random_sample()
      bucket_id = min([i for i in xrange(len(train_buckets_scale))
                       if train_buckets_scale[i] > random_number_01])

      # Get a batch and make a step.
      start_time = time.time()
      encoder_inputs, decoder_inputs, target_weights = model.get_batch(
          train_set, bucket_id)
      # print("encoder_inputs", "-"*80)
      # print(encoder_inputs)
      # print("decoder_inputs", "-"*80)
      # print(decoder_inputs)
      # print("bucket_id", bucket_id)
      _, step_loss, _ = model.step(sess, encoder_inputs, decoder_inputs,
                                   target_weights, bucket_id, False)
      step_time += (time.time() - start_time) / FLAGS.steps_per_checkpoint
      loss += step_loss / FLAGS.steps_per_checkpoint
      current_step += 1
      # print("loss", loss)
      sys.stdout.flush()

      # Once in a while, we save checkpoint, print statistics, and run evals.
      if current_step % FLAGS.steps_per_checkpoint == 0:
        # Print statistics for the previous epoch.
        perplexity = math.exp(loss) if loss < 300 else float('inf')
        print ("global step %d learning rate %.4f step-time %.2f perplexity "
               "%.2f" % (model.global_step.eval(), model.learning_rate.eval(),
                         step_time, perplexity))
        # Decrease learning rate if no improvement was seen over last 3 times.
        if len(previous_losses) > 2 and loss > max(previous_losses[-3:]):
          sess.run(model.learning_rate_decay_op)
        previous_losses.append(loss)
        # Save checkpoint and zero timer and loss.
        checkpoint_path = os.path.join(FLAGS.train_dir, "translate.ckpt")
        model.saver.save(sess, checkpoint_path, global_step=model.global_step)
        step_time, loss = 0.0, 0.0
        # Run evals on development set and print their perplexity.
        for bucket_id in xrange(len(_buckets)):
          encoder_inputs, decoder_inputs, target_weights = model.get_batch(
              dev_set, bucket_id)
          _, eval_loss, _ = model.step(sess, encoder_inputs, decoder_inputs,
                                       target_weights, bucket_id, True)
          eval_ppx = math.exp(eval_loss) if eval_loss < 300 else float('inf')
          print("  eval: bucket %d perplexity %.2f" % (bucket_id, eval_ppx))
        sys.stdout.flush()

Example 110

Project: facenet
Source File: mnist_center_loss.py
View license
def main(argv=None):  # pylint: disable=unused-argument
    if FLAGS.self_test:
        print('Running self-test.')
        train_data, train_labels = fake_data(256)
        validation_data, validation_labels = fake_data(EVAL_BATCH_SIZE)
        test_data, test_labels = fake_data(EVAL_BATCH_SIZE)
        num_epochs = 1
    else:
        # Get the data.
        train_data_filename = maybe_download('train-images-idx3-ubyte.gz')
        train_labels_filename = maybe_download('train-labels-idx1-ubyte.gz')
        test_data_filename = maybe_download('t10k-images-idx3-ubyte.gz')
        test_labels_filename = maybe_download('t10k-labels-idx1-ubyte.gz')
    
        # Extract it into numpy arrays.
        train_data = extract_data(train_data_filename, 60000)
        train_labels = extract_labels(train_labels_filename, 60000)
        test_data = extract_data(test_data_filename, 10000)
        test_labels = extract_labels(test_labels_filename, 10000)
    
        # Generate a validation set.
        validation_data = train_data[:VALIDATION_SIZE, ...]
        validation_labels = train_labels[:VALIDATION_SIZE]
        train_data = train_data[VALIDATION_SIZE:, ...]
        train_labels = train_labels[VALIDATION_SIZE:]
        num_epochs = NUM_EPOCHS
    train_size = train_labels.shape[0]

    # This is where training samples and labels are fed to the graph.
    # These placeholder nodes will be fed a batch of training data at each
    # training step using the {feed_dict} argument to the Run() call below.
    train_data_node = tf.placeholder(
        data_type(),
        shape=(BATCH_SIZE, IMAGE_SIZE, IMAGE_SIZE, NUM_CHANNELS))
    train_labels_node = tf.placeholder(tf.int64, shape=(BATCH_SIZE,))
    eval_data = tf.placeholder(
        data_type(),
        shape=(EVAL_BATCH_SIZE, IMAGE_SIZE, IMAGE_SIZE, NUM_CHANNELS))

    # The variables below hold all the trainable weights. They are passed an
    # initial value which will be assigned when we call:
    # {tf.initialize_all_variables().run()}
    conv1_weights = tf.Variable(
        tf.truncated_normal([5, 5, NUM_CHANNELS, 32],  # 5x5 filter, depth 32.
                            stddev=0.1,
                            seed=SEED, dtype=data_type()))
    conv1_biases = tf.Variable(tf.zeros([32], dtype=data_type()))
    conv2_weights = tf.Variable(tf.truncated_normal(
        [5, 5, 32, 64], stddev=0.1,
        seed=SEED, dtype=data_type()))
    conv2_biases = tf.Variable(tf.constant(0.1, shape=[64], dtype=data_type()))
    fc1_weights = tf.Variable(  # fully connected, depth 512.
        tf.truncated_normal([IMAGE_SIZE // 4 * IMAGE_SIZE // 4 * 64, 512],
                            stddev=0.1,
                            seed=SEED,
                            dtype=data_type()))
    fc1_biases = tf.Variable(tf.constant(0.1, shape=[512], dtype=data_type()))
    fc1p_weights = tf.Variable(  # fully connected, depth 512.
        tf.truncated_normal([512, 2],
                            stddev=0.1,
                            seed=SEED,
                            dtype=data_type()))
    fc1p_biases = tf.Variable(tf.constant(0.1, shape=[2], dtype=data_type()))
    fc2_weights = tf.Variable(tf.truncated_normal([2, NUM_LABELS],
                                                  stddev=0.1,
                                                  seed=SEED,
                                                  dtype=data_type()))
    fc2_biases = tf.Variable(tf.constant(
        0.1, shape=[NUM_LABELS], dtype=data_type()))
    
    def batch_norm(x, phase_train):  #pylint: disable=unused-variable
        """
        Batch normalization on convolutional maps.
        Args:
            x:           Tensor, 4D BHWD input maps
            n_out:       integer, depth of input maps
            phase_train: boolean tf.Variable, true indicates training phase
            scope:       string, variable scope
            affn:      whether to affn-transform outputs
        Return:
            normed:      batch-normalized maps
        Ref: http://stackoverflow.com/questions/33949786/how-could-i-use-batch-normalization-in-tensorflow/33950177
        """
        name = 'batch_norm'
        with tf.variable_scope(name):
            phase_train = tf.convert_to_tensor(phase_train, dtype=tf.bool)
            n_out = int(x.get_shape()[-1])
            beta = tf.Variable(tf.constant(0.0, shape=[n_out], dtype=x.dtype),
                               name=name+'/beta', trainable=True, dtype=x.dtype)
            gamma = tf.Variable(tf.constant(1.0, shape=[n_out], dtype=x.dtype),
                                name=name+'/gamma', trainable=True, dtype=x.dtype)
          
            batch_mean, batch_var = tf.nn.moments(x, [0], name='moments')
            ema = tf.train.ExponentialMovingAverage(decay=0.9)
            def mean_var_with_update():
                ema_apply_op = ema.apply([batch_mean, batch_var])
                with tf.control_dependencies([ema_apply_op]):
                    return tf.identity(batch_mean), tf.identity(batch_var)
            mean, var = control_flow_ops.cond(phase_train,
                                              mean_var_with_update,
                                              lambda: (ema.average(batch_mean), ema.average(batch_var)))
            normed = tf.nn.batch_normalization(x, mean, var, beta, gamma, 1e-3)
        return normed
    

    # We will replicate the model structure for the training subgraph, as well
    # as the evaluation subgraphs, while sharing the trainable parameters.
    def model(data, train=False):
        """The Model definition."""
        # 2D convolution, with 'SAME' padding (i.e. the output feature map has
        # the same size as the input). Note that {strides} is a 4D array whose
        # shape matches the data layout: [image index, y, x, depth].
        conv = tf.nn.conv2d(data,
                            conv1_weights,
                            strides=[1, 1, 1, 1],
                            padding='SAME')
        # Bias and rectified linear non-linearity.
        relu = tf.nn.relu(tf.nn.bias_add(conv, conv1_biases))
        # Max pooling. The kernel size spec {ksize} also follows the layout of
        # the data. Here we have a pooling window of 2, and a stride of 2.
        pool = tf.nn.max_pool(relu,
                              ksize=[1, 2, 2, 1],
                              strides=[1, 2, 2, 1],
                              padding='SAME')
        conv = tf.nn.conv2d(pool,
                            conv2_weights,
                            strides=[1, 1, 1, 1],
                            padding='SAME')
        relu = tf.nn.relu(tf.nn.bias_add(conv, conv2_biases))
        pool = tf.nn.max_pool(relu,
                              ksize=[1, 2, 2, 1],
                              strides=[1, 2, 2, 1],
                              padding='SAME')
        # Reshape the feature map cuboid into a 2D matrix to feed it to the
        # fully connected layers.
        pool_shape = pool.get_shape().as_list() #pylint: disable=no-member
        reshape = tf.reshape(
            pool,
            [pool_shape[0], pool_shape[1] * pool_shape[2] * pool_shape[3]])
        # Fully connected layer. Note that the '+' operation automatically
        # broadcasts the biases.
        hidden = tf.nn.relu(tf.matmul(reshape, fc1_weights) + fc1_biases)
        # Add a 50% dropout during training only. Dropout also scales
        # activations such that no rescaling is needed at evaluation time.
        if train:
            hidden = tf.nn.dropout(hidden, 0.5, seed=SEED)

        hidden = tf.matmul(hidden, fc1p_weights) + fc1p_biases

        return tf.nn.relu(tf.matmul(hidden, fc2_weights) + fc2_biases), hidden

    def center_loss_op(logits, labels):
        alfa = 0.05
        nrof_features = logits.get_shape()[1]
        centers = tf.get_variable('centers', shape=(nrof_features), dtype=tf.float32,
            initializer=tf.constant_initializer(value=0.0, dtype=tf.float32), trainable=False)
        # Define center loss
        #center_loss = tf.reduce_sum(tf.pow(tf.abs(logits - centers), 2.0))
        center_loss = tf.nn.l2_loss(logits - centers)
        one_hot = tf.one_hot(labels, nrof_features, axis=1, dtype=tf.float32, name='one_hot')
        delta1 = tf.reduce_mean((centers-logits)*one_hot,0)
        delta2 = 1+tf.reduce_mean(one_hot,0)
        centers_delta = delta1 / delta2
        update_centers = tf.assign_add(centers, -alfa*centers_delta)
        return center_loss, update_centers
  
    # Training computation: logits + cross-entropy loss.
    logits, hidden = model(train_data_node, True)
    #logits = batch_norm(logits, True)
    xent_loss = tf.reduce_mean(tf.nn.sparse_softmax_cross_entropy_with_logits(
        logits, train_labels_node))
    beta = 1e-6
    center_loss, update_centers = center_loss_op(hidden, train_labels_node)
    loss = xent_loss + beta * center_loss
  
    # L2 regularization for the fully connected parameters.
    regularizers = (tf.nn.l2_loss(fc1_weights) + tf.nn.l2_loss(fc1_biases) +
                    tf.nn.l2_loss(fc2_weights) + tf.nn.l2_loss(fc2_biases))
    # Add the regularization term to the loss.
    loss += 5e-4 * regularizers
  
    # Optimizer: set up a variable that's incremented once per batch and
    # controls the learning rate decay.
    batch = tf.Variable(0, dtype=data_type())
    # Decay once per epoch, using an exponential schedule starting at 0.01.
    learning_rate = tf.train.exponential_decay(
        0.01,                # Base learning rate.
        batch * BATCH_SIZE,  # Current index into the dataset.
        train_size,          # Decay step.
        0.95,                # Decay rate.
        staircase=True)
    # Use simple momentum for the optimization.
    optimizer = tf.train.MomentumOptimizer(learning_rate,
                                           0.9).minimize(loss,
                                                         global_step=batch)
  
    # Predictions for the current training minibatch.
    train_prediction = tf.nn.softmax(logits)
  
    # Predictions for the test and validation, which we'll compute less often.
    eval_logits, eval_embeddings = model(eval_data)
    eval_prediction = tf.nn.softmax(eval_logits)
    
    # Small utility function to evaluate a dataset by feeding batches of data to
    # {eval_data} and pulling the results from {eval_predictions}.
    # Saves memory and enables this to run on smaller GPUs.
    def eval_in_batches(data, sess):
        """Get all predictions for a dataset by running it in small batches."""
        size = data.shape[0]
        if size < EVAL_BATCH_SIZE:
            raise ValueError("batch size for evals larger than dataset: %d" % size)
        predictions = np.ndarray(shape=(size, NUM_LABELS), dtype=np.float32)
        for begin in xrange(0, size, EVAL_BATCH_SIZE):
            end = begin + EVAL_BATCH_SIZE
            if end <= size:
                predictions[begin:end, :] = sess.run(
                    eval_prediction,
                    feed_dict={eval_data: data[begin:end, ...]})
            else:
                batch_predictions = sess.run(
                    eval_prediction,
                    feed_dict={eval_data: data[-EVAL_BATCH_SIZE:, ...]})
                predictions[begin:, :] = batch_predictions[begin - size:, :]
        return predictions
  
    def calculate_embeddings(data, sess):
        """Get all predictions for a dataset by running it in small batches."""
        size = data.shape[0]
        if size < EVAL_BATCH_SIZE:
            raise ValueError("batch size for evals larger than dataset: %d" % size)
        predictions = np.ndarray(shape=(size, 2), dtype=np.float32)
        for begin in xrange(0, size, EVAL_BATCH_SIZE):
            end = begin + EVAL_BATCH_SIZE
            if end <= size:
                predictions[begin:end, :] = sess.run(
                    eval_embeddings,
                    feed_dict={eval_data: data[begin:end, ...]})
            else:
                batch_predictions = sess.run(
                    eval_embeddings,
                    feed_dict={eval_data: data[-EVAL_BATCH_SIZE:, ...]})
                predictions[begin:, :] = batch_predictions[begin - size:, :]
        return predictions

    # Create a local session to run the training.
    start_time = time.time()
    with tf.Session() as sess:
        # Run all the initializers to prepare the trainable parameters.
        tf.initialize_all_variables().run() #pylint: disable=no-member
        print('Initialized!')
        # Loop through training steps.
        for step in xrange(int(num_epochs * train_size) // BATCH_SIZE):
            # Compute the offset of the current minibatch in the data.
            # Note that we could use better randomization across epochs.
            offset = (step * BATCH_SIZE) % (train_size - BATCH_SIZE)
            batch_data = train_data[offset:(offset + BATCH_SIZE), ...]
            batch_labels = train_labels[offset:(offset + BATCH_SIZE)]
            # This dictionary maps the batch data (as a numpy array) to the
            # node in the graph it should be fed to.
            feed_dict = {train_data_node: batch_data,
                         train_labels_node: batch_labels}
            # Run the graph and fetch some of the nodes.
            #_, l, lr, predictions = sess.run([optimizer, loss, learning_rate, train_prediction], feed_dict=feed_dict)
            _, _, cl, l, lr, predictions = sess.run([update_centers, optimizer, center_loss, loss, learning_rate, train_prediction], feed_dict=feed_dict)
            if step % EVAL_FREQUENCY == 0:
                elapsed_time = time.time() - start_time
                start_time = time.time()
                print('Step %d (epoch %.2f), %.1f ms' %
                      (step, float(step) * BATCH_SIZE / train_size,
                       1000 * elapsed_time / EVAL_FREQUENCY))
                print('Minibatch loss: %.3f  %.3f, learning rate: %.6f' % (l, cl*beta, lr))
                print('Minibatch error: %.1f%%' % error_rate(predictions, batch_labels))
                print('Validation error: %.1f%%' % error_rate(
                    eval_in_batches(validation_data, sess), validation_labels))
                sys.stdout.flush()
        # Finally print the result!
        test_error = error_rate(eval_in_batches(test_data, sess), test_labels)
        print('Test error: %.1f%%' % test_error)
        if FLAGS.self_test:
            print('test_error', test_error)
            assert test_error == 0.0, 'expected 0.0 test_error, got %.2f' % (
                test_error,)
            
        train_embeddings = calculate_embeddings(train_data, sess)
        
        color_list = ['b', 'g', 'r', 'c', 'm', 'y', 'k', 'b', 'g', 'r', 'c' ]
        plt.figure(1)
        for n in range(0,10):
            idx = np.where(train_labels[0:10000]==n)
            plt.plot(train_embeddings[idx,0], train_embeddings[idx,1], color_list[n]+'.')
        plt.show()

Example 111

Project: TrustRouter
Source File: test_re.py
View license
def run_re_tests():
    from test.re_tests import tests, SUCCEED, FAIL, SYNTAX_ERROR
    if verbose:
        print('Running re_tests test suite')
    else:
        # To save time, only run the first and last 10 tests
        #tests = tests[:10] + tests[-10:]
        pass

    for t in tests:
        sys.stdout.flush()
        pattern = s = outcome = repl = expected = None
        if len(t) == 5:
            pattern, s, outcome, repl, expected = t
        elif len(t) == 3:
            pattern, s, outcome = t
        else:
            raise ValueError('Test tuples should have 3 or 5 fields', t)

        try:
            obj = re.compile(pattern)
        except re.error:
            if outcome == SYNTAX_ERROR: pass  # Expected a syntax error
            else:
                print('=== Syntax error:', t)
        except KeyboardInterrupt: raise KeyboardInterrupt
        except:
            print('*** Unexpected error ***', t)
            if verbose:
                traceback.print_exc(file=sys.stdout)
        else:
            try:
                result = obj.search(s)
            except re.error as msg:
                print('=== Unexpected exception', t, repr(msg))
            if outcome == SYNTAX_ERROR:
                # This should have been a syntax error; forget it.
                pass
            elif outcome == FAIL:
                if result is None: pass   # No match, as expected
                else: print('=== Succeeded incorrectly', t)
            elif outcome == SUCCEED:
                if result is not None:
                    # Matched, as expected, so now we compute the
                    # result string and compare it to our expected result.
                    start, end = result.span(0)
                    vardict={'found': result.group(0),
                             'groups': result.group(),
                             'flags': result.re.flags}
                    for i in range(1, 100):
                        try:
                            gi = result.group(i)
                            # Special hack because else the string concat fails:
                            if gi is None:
                                gi = "None"
                        except IndexError:
                            gi = "Error"
                        vardict['g%d' % i] = gi
                    for i in result.re.groupindex.keys():
                        try:
                            gi = result.group(i)
                            if gi is None:
                                gi = "None"
                        except IndexError:
                            gi = "Error"
                        vardict[i] = gi
                    repl = eval(repl, vardict)
                    if repl != expected:
                        print('=== grouping error', t, end=' ')
                        print(repr(repl) + ' should be ' + repr(expected))
                else:
                    print('=== Failed incorrectly', t)

                # Try the match with both pattern and string converted to
                # bytes, and check that it still succeeds.
                try:
                    bpat = bytes(pattern, "ascii")
                    bs = bytes(s, "ascii")
                except UnicodeEncodeError:
                    # skip non-ascii tests
                    pass
                else:
                    try:
                        bpat = re.compile(bpat)
                    except Exception:
                        print('=== Fails on bytes pattern compile', t)
                        if verbose:
                            traceback.print_exc(file=sys.stdout)
                    else:
                        bytes_result = bpat.search(bs)
                        if bytes_result is None:
                            print('=== Fails on bytes pattern match', t)

                # Try the match with the search area limited to the extent
                # of the match and see if it still succeeds.  \B will
                # break (because it won't match at the end or start of a
                # string), so we'll ignore patterns that feature it.

                if pattern[:2] != '\\B' and pattern[-2:] != '\\B' \
                               and result is not None:
                    obj = re.compile(pattern)
                    result = obj.search(s, result.start(0), result.end(0) + 1)
                    if result is None:
                        print('=== Failed on range-limited match', t)

                # Try the match with IGNORECASE enabled, and check that it
                # still succeeds.
                obj = re.compile(pattern, re.IGNORECASE)
                result = obj.search(s)
                if result is None:
                    print('=== Fails on case-insensitive match', t)

                # Try the match with LOCALE enabled, and check that it
                # still succeeds.
                if '(?u)' not in pattern:
                    obj = re.compile(pattern, re.LOCALE)
                    result = obj.search(s)
                    if result is None:
                        print('=== Fails on locale-sensitive match', t)

                # Try the match with UNICODE locale enabled, and check
                # that it still succeeds.
                obj = re.compile(pattern, re.UNICODE)
                result = obj.search(s)
                if result is None:
                    print('=== Fails on unicode-sensitive match', t)

Example 112

Project: GroundedTranslation
Source File: generate.py
View license
    def generate_sentences(self, filepath, val=True):
        """
        Generates descriptions of images for --generation_timesteps
        iterations through the LSTM. Each input description is clipped to
        the first <BOS> token, or, if --generate_from_N_words is set, to the
        first N following words (N + 1 BOS token).
        This process can be additionally conditioned
        on source language hidden representations, if provided by the
        --source_vectors parameter.
        The output is clipped to the first EOS generated, if it exists.

        TODO: duplicated method with generate.py
        """

        if self.args.beam_width > 1:
            prefix = "val" if val else "test"
            handle = codecs.open("%s/%sGenerated" % (filepath, prefix), "w",
                                 'utf-8')
            logger.info("Generating %s descriptions", prefix)

            start_gen = self.args.generate_from_N_words  # Default 0
            start_gen = start_gen + 1  # include BOS

            generator = self.data_gen.generation_generator(prefix, batch_size=1)
            seen = 0
            # we are going to beam search for the most probably sentence.
            # let's do this one sentence at a time to make the logging output
            # easier to understand
            for data in generator:
                text = data[0]['text']
                # Append the first start_gen words to the complete_sentences list
                # for each instance in the batch.
                complete_sentences = [[] for _ in range(text.shape[0])]
                for t in range(start_gen):  # minimum 1
                    for i in range(text.shape[0]):
                        w = np.argmax(text[i, t])
                        complete_sentences[i].append(self.index2word[w])
                del data[0]['text']
                text = self.reset_text_arrays(text, start_gen)
                Y_target = data[1]['output']
                data[0]['text'] = text

                max_beam_width = self.args.beam_width
                structs = self.make_duplicate_matrices(data[0], max_beam_width)

                # A beam is a 2-tuple with the probability of the sequence and
                # the words in that sequence. Start with empty beams
                beams = [(0.0, [])]
                # collects beams that are in the top candidates and 
                # emitted a <E> token.
                finished = [] 
                for t in range(start_gen, self.args.generation_timesteps):
                    # Store the candidates produced at timestep t, will be
                    # pruned at the end of the timestep
                    candidates = []

                    # we take a view of the datastructures, which means we're only
                    # ever generating a prediction for the next word. This saves a
                    # lot of cycles.
                    preds = self.model.predict(structs, verbose=0)

                    # The last indices in preds are the predicted words
                    next_word_indices = preds[:, t-1]
                    sorted_indices = np.argsort(-next_word_indices, axis=1)

                    # Each instance in structs is holding the history of a
                    # beam, and so there is a direct connection between the
                    # index of a beam in beams and the index of an instance in
                    # structs.
                    for beam_idx, b in enumerate(beams):
                        # get the sorted predictions for the beam_idx'th beam
                        beam_predictions = sorted_indices[beam_idx]
                        for top_idx in range(self.args.beam_width):
                            wordIndex = beam_predictions[top_idx]
                            wordProb = next_word_indices[beam_idx][beam_predictions[top_idx]]
                            # For the beam_idxth beam, add the log probability
                            # of the top_idxth predicted word to the previous
                            # log probability of the sequence, and  append the 
                            # top_idxth predicted word to the sequence of words 
                            candidates.append([b[0] + math.log(wordProb), b[1] + [wordIndex]])

                    candidates.sort(reverse = True)
                    if self.args.verbose:
                        logger.info("Candidates in the beam")
                        logger.info("---")
                        for c in candidates:
                            logger.info(" ".join([self.index2word[x] for x in c[1]]) + " (%f)" % c[0])

                    beams = candidates[:max_beam_width] # prune the beams
                    pruned = []
                    for b in beams:
                        # If a top candidate emitted an EOS token then 
                        # a) add it to the list of finished sequences
                        # b) remove it from the beams and decrease the 
                        # maximum size of the beams.
                        if b[1][-1] == self.word2index["<E>"]:
                            finished.append(b)
                            if max_beam_width >= 1:
                                max_beam_width -= 1
                        else:
                            pruned.append(b)
                    
                    beams = pruned[:max_beam_width]

                    if self.args.verbose:
                        logger.info("Pruned beams")
                        logger.info("---")
                        for b in beams:
                            logger.info(" ".join([self.index2word[x] for x in b[1]]) + "(%f)" % b[0])

                    if max_beam_width == 0:
                        # We have sampled max_beam_width sequences with an <E>
                        # token so stop the beam search.
                        break

                    # Reproduce the structs for the beam search so we can keep
                    # track of the state of each beam
                    structs = self.make_duplicate_matrices(data[0], max_beam_width)

                    # Rewrite the 1-hot word features with the
                    # so-far-predcicted tokens in a beam.
                    for bidx, b in enumerate(beams):
                        for idx, w in enumerate(b[1]):
                            next_word_index = w
                            structs['text'][bidx, idx+1, w] = 1.

                # If none of the sentences emitted an <E> token while
                # decoding, add the final beams into the final candidates
                if len(finished) == 0:
                    for leftover in beams:
                        finished.append(leftover)

                # Normalise the probabilities by the length of the sequences
                # as suggested by Graves (2012) http://arxiv.org/abs/1211.3711
                for f in finished:
                    f[0] = f[0] / len(f[1])
                finished.sort(reverse=True)

                if self.args.verbose:
                    logger.info("Length-normalised samples")
                    logger.info("---")
                    for f in finished:
                        logger.info(" ".join([self.index2word[x] for x in f[1]]) + "(%f)" % f[0])

                # Emit the lowest (log) probability sequence
                best_beam = finished[0]
                complete_sentences[i] = [self.index2word[x] for x in best_beam[1]]
                handle.write(' '.join([x for x
                                       in itertools.takewhile(
                                           lambda n: n != "<E>", complete_sentences[i])]) + "\n")
                if self.args.verbose:
                    logger.info("%s (%f)",' '.join([x for x
                                          in itertools.takewhile(
                                              lambda n: n != "<E>",
                                              complete_sentences[i])]),
                                          best_beam[0])

                seen += text.shape[0]
                if seen == self.data_gen.split_sizes['val']:
                    # Hacky way to break out of the generator
                    break
            handle.close()
        else:
            # We are going to arg max decode a sequence.
            prefix = "val" if val else "test"
            logger.info("Generating %s descriptions", prefix)
            start_gen = self.args.generate_from_N_words + 1  # include BOS
            handle = codecs.open("%s/%sGenerated" % (filepath, prefix), 
                                 "w", 'utf-8')

            generator = self.data_gen.generation_generator(prefix)
            seen = 0
            for data in generator:
                text = deepcopy(data[0]['text'])
                # Append the first start_gen words to the complete_sentences list
                # for each instance in the batch.
                complete_sentences = [[] for _ in range(text.shape[0])]
                for t in range(start_gen):  # minimum 1
                    for i in range(text .shape[0]):
                        w = np.argmax(text[i, t])
                        complete_sentences[i].append(self.index2word[w])
                del data[0]['text']
                text = self.reset_text_arrays(text, start_gen)
                Y_target = data[1]['output']
                data[0]['text'] = text

                for t in range(start_gen, self.args.generation_timesteps):
                    logger.debug("Input token: %s" % self.index2word[np.argmax(text[0,t-1])])
                    preds = self.model.predict(data[0],
                                               verbose=0)

                    # Look at the last indices for the words.
                    next_word_indices = np.argmax(preds[:, t-1], axis=1)
                    logger.debug("Predicted token: %s" % self.index2word[next_word_indices[0]])
                    # update array[0]/sentence-so-far with generated words.
                    for i in range(len(next_word_indices)):
                        data[0]['text'][i, t, next_word_indices[i]] = 1.
                    next_words = [self.index2word[x] for x in next_word_indices]
                    for i in range(len(next_words)):
                        complete_sentences[i].append(next_words[i])

                sys.stdout.flush()
                # print/extract each sentence until it hits the first end-of-string token
                for s in complete_sentences:
                    if self.args.verbose:
                        logger.info("%s",' '.join([x for x
                                              in itertools.takewhile(
                                                  lambda n: n != "<E>",
                                                  complete_sentences[i])]))
                    decoded_str = ' '.join([x for x
                                            in itertools.takewhile(
                                                lambda n: n != "<E>", s[1:])])
                    handle.write(decoded_str + "\n")

                seen += text.shape[0]
                if seen == self.data_gen.split_sizes[prefix]:
                    # Hacky way to break out of the generator
                    break
            handle.close()

Example 113

Project: Wallace
Source File: tests.py
View license
        def test_run_rogers(self):

            """
            SIMULATE ROGERS
            """

            hit_id = str(random.random())

            overall_start_time = timenow()

            print("Running simulated experiment...", end="\r")
            sys.stdout.flush()

            exp_setup_start = timenow()
            exp = RogersExperiment(self.db)
            exp_setup_stop = timenow()

            exp_setup_start2 = timenow()
            exp = RogersExperiment(self.db)
            exp_setup_stop2 = timenow()

            p_ids = []
            p_times = []
            dum = timenow()
            assign_time = dum - dum
            process_time = dum - dum

            while exp.networks(full=False):

                num_completed_participants = len(exp.networks()[0].nodes(type=Agent))

                if p_times:
                    print("Running simulated experiment... participant {} of {}, {} participants failed. Prev time: {}".format(
                        num_completed_participants+1,
                        exp.networks()[0].max_size,
                        len(exp.networks()[0].nodes(failed=True)),
                        p_times[-1]),
                        end="\r")
                else:
                    print("Running simulated experiment... participant {} of {}, {} participants failed.".format(
                        num_completed_participants+1,
                        exp.networks()[0].max_size,
                        len(exp.networks()[0].nodes(failed=True))),
                        end="\r")
                sys.stdout.flush()

                worker_id = str(random.random())
                assignment_id = str(random.random())
                from models import Participant
                p = Participant(workerid=worker_id, assignmentid=assignment_id, hitid=hit_id)
                self.db.add(p)
                self.db.commit()
                p_id = p.uniqueid
                p_ids.append(p_id)
                p_start_time = timenow()

                while True:
                    assign_start_time = timenow()
                    network = exp.get_network_for_participant(participant=p)
                    if network is None:
                        break
                    else:
                        agent = exp.create_node(
                            participant_id=p_id,
                            network=network)
                        exp.add_node_to_network(
                            participant_id=p_id,
                            node=agent,
                            network=network)
                        self.db.commit()
                        exp.node_post_request(participant_id=p_id, node=agent)
                        self.db.commit()
                        assign_stop_time = timenow()
                        assign_time += (assign_stop_time - assign_start_time)

                        process_start_time = timenow()
                        agent.receive()
                        from operator import attrgetter
                        current_state = max(State.query.filter_by(network_id=agent.network_id).all(), key=attrgetter('creation_time')).contents
                        if float(current_state) >= 0.5:
                            right_answer = "blue"
                            wrong_answer = "yellow"
                        else:
                            right_answer = "yellow"
                            wrong_answer = "blue"
                        if num_completed_participants == 0:
                            info = Meme(origin=agent, contents=right_answer)
                        else:
                            if random.random() < 0.9:
                                info = Meme(origin=agent, contents=right_answer)
                            else:
                                info = Meme(origin=agent, contents=wrong_answer)
                        self.db.commit()
                        exp.info_post_request(
                            node=agent,
                            info=info)
                        #print("state: {}, answer: {}, score: {}, fitness {}".format(current_state, info.contents, agent.score, agent.fitness))
                        process_stop_time = timenow()
                        process_time += (process_stop_time - process_start_time)

                worked = exp.data_check(participant=p)
                assert worked
                bonus = exp.bonus(participant=p)
                assert bonus >= 0
                assert bonus <= 1
                attended = exp.attention_check(participant=p)
                if not attended:

                    participant_nodes = models.Node.query\
                        .filter_by(participant_id=p_id, failed=False)\
                        .all()
                    p.status = 102

                    for node in participant_nodes:
                        node.fail()

                    self.db.commit()
                else:
                    exp.submission_successful(participant=p)

                p_stop_time = timenow()
                p_times.append(p_stop_time - p_start_time)

            print("Running simulated experiment...      done!                                      ")
            sys.stdout.flush()

            overall_stop_time = timenow()

            assert len(exp.networks()) == exp.practice_repeats + exp.experiment_repeats

            """
            TEST NODES
            """

            print("Testing nodes...", end="\r")
            sys.stdout.flush()

            for network in [exp.networks()[0]]:

                agents = network.nodes(type=Agent)
                assert len(agents) == network.max_size

                source = network.nodes(type=Source)
                assert len(source) == 1
                source = source[0]
                assert type(source) == RogersSource

                environment = network.nodes(type=Environment)
                assert len(environment) == 1
                environment = environment[0]
                assert type(environment) == RogersEnvironment

                vectors = network.vectors()

                role = network.role
                if role == "practice":
                    for agent in agents:
                        assert type(agent) == RogersAgentFounder
                elif role == "catch":
                    for agent in agents:
                        assert type(agent) == RogersAgentFounder
                else:
                    for agent in agents:
                        if agent.generation == 0:
                            assert type(agent) == RogersAgentFounder
                        else:
                            assert type(agent) == RogersAgent

                for agent in agents:
                    if agent.generation == 0:
                        assert len(agent.vectors(direction="incoming")) == 2
                        assert agent.is_connected(direction="from", whom=source)
                        assert agent.is_connected(direction="from", whom=environment)
                    else:
                        assert len(agent.vectors(direction="incoming")) in [2, 3]
                        assert not agent.is_connected(direction="from", whom=source)
                        assert agent.is_connected(direction="from", whom=environment)
                        assert RogersAgent in [type(a) for a in agent.neighbors(direction="from")] or\
                            RogersAgentFounder in [type(a) for a in agent.neighbors(direction="from")]

            print("Testing nodes...                     done!")
            sys.stdout.flush()

            """
            TEST VECTORS
            """

            print("Testing vectors...", end="\r")
            sys.stdout.flush()

            for network in [exp.networks()[0]]:

                agents = network.nodes(type=Agent)
                vectors = network.vectors()
                source = network.nodes(type=Source)[0]
                environment = network.nodes(type=Environment)[0]

                for v in vectors:
                    if isinstance(v.origin, Agent):
                        assert v.origin.generation == v.destination.generation - 1
                    else:
                        assert isinstance(v.origin, Source) or isinstance(v.origin, Environment)

                for agent in agents:
                    if agent.generation == 0:
                        assert len(models.Vector.query.filter_by(origin_id=source.id, destination_id=agent.id).all()) == 1
                    else:
                        assert len(models.Vector.query.filter_by(origin_id=source.id, destination_id=agent.id).all()) == 0

                for agent in agents:
                    assert len([v for v in vectors if v.origin_id == environment.id and v.destination_id == agent.id]) == 1

                for v in [v for v in vectors if v.origin_id == source.id]:
                    assert isinstance(v.destination, RogersAgentFounder)

            print("Testing vectors...                   done!")
            sys.stdout.flush()

            """
            TEST INFOS
            """

            print("Testing infos...", end="\r")
            sys.stdout.flush()

            for network in [exp.networks()[0]]:

                agents = network.nodes(type=Agent)
                vectors = network.vectors()
                source = network.nodes(type=Source)[0]
                environment = network.nodes(type=Environment)[0]
                infos = network.infos()

                for agent in agents:
                    assert len([i for i in infos if i.origin_id == agent.id]) == 2
                    assert len([i for i in infos if i.origin_id == agent.id and isinstance(i, Gene)]) == 1
                    assert len([i for i in infos if i.origin_id == agent.id and isinstance(i, LearningGene)]) == 1
                    assert len([i for i in infos if i.origin_id == agent.id and isinstance(i, Meme)]) == 1

            print("Testing infos...                     done!")
            sys.stdout.flush()

            """
            TEST TRANSMISSIONS
            """

            print("Testing transmissions...", end="\r")
            sys.stdout.flush()

            for network in [exp.networks()[0]]:

                agents = network.nodes(type=Agent)
                vectors = network.vectors()
                source = network.nodes(type=Source)[0]
                environment = network.nodes(type=Environment)[0]
                infos = network.infos()
                transmissions = network.transmissions()

                for agent in agents:
                    in_ts = [t for t in transmissions if t.destination_id == agent.id]
                    types = [type(t.info) for t in in_ts]

                    assert len(in_ts) == 2
                    assert len([t for t in transmissions if t.destination_id == agent.id and t.status == "pending"]) == 0

                    lg = [i for i in infos if i.origin_id == agent.id and isinstance(i, LearningGene)]
                    assert len(lg) == 1
                    lg = lg[0]

                    if lg.contents == "asocial":
                        assert State in types
                        assert LearningGene in types
                        assert Meme not in types
                    else:
                        assert State not in types
                        assert LearningGene in types
                        assert Meme in types

            print("Testing transmissions...             done!")

            """
            TEST FITNESS
            """

            print("Testing fitness...", end="\r")
            sys.stdout.flush()

            p0_nodes = models.Node.query.filter_by(participant_id=p_ids[0]).all()

            assert len(p0_nodes) == len(exp.networks())

            is_asocial = True
            e = 2
            b = 1
            c = 0.3*b
            baseline = c+0.0001

            for n in p0_nodes:
                assert n.fitness == (baseline + 1 * b - is_asocial * c) ** e

            for network in [exp.networks()[0]]:

                agents = network.nodes(type=Agent)

                for agent in agents:
                    is_asocial = agent.infos(type=LearningGene)[0].contents == "asocial"
                    assert agent.fitness == ((baseline + agent.score*b - is_asocial*c) ** e)

            print("Testing fitness...                   done!")
            sys.stdout.flush()

            """
            TEST BONUS
            """

            print("Testing bonus payments...", end="\r")
            sys.stdout.flush()

            assert exp.bonus(participant=Participant.query.filter_by(uniqueid=p_ids[0]).all()[0]) == exp.bonus_payment

            print("Testing bonus payments...            done!")
            sys.stdout.flush()

            print("All tests passed: good job!")

            print("Timings:")
            overall_time = overall_stop_time - overall_start_time
            print("Overall time to simulate experiment: {}".format(overall_time))
            setup_time = exp_setup_stop - exp_setup_start
            print("Experiment setup(): {}".format(setup_time))
            print("Experiment load: {}".format(exp_setup_stop2 - exp_setup_start2))
            print("Participant assignment: {}".format(assign_time))
            print("Participant processing: {}".format(process_time))
            for i in range(len(p_times)):
                if i == 0:
                    total_time = p_times[i]
                else:
                    total_time += p_times[i]
                print("Participant {}: {}, total: {}".format(i, p_times[i], total_time))

            print("#########")
            test = [p.total_seconds() for p in p_times]
            print(test)

Example 114

Project: EQcorrscan
Source File: lag_calc.py
View license
def lag_calc(detections, detect_data, template_names, templates,
             shift_len=0.2, min_cc=0.4, cores=1, interpolate=False,
             plot=False, parallel=True, debug=0):
    """
    Main lag-calculation function for detections of specific events.

    Overseer function to take a list of detection objects, cut the data for
    them to lengths of the same length of the template + shift_len on
    either side. This will output a :class:`obspy.core.event.Catalog` of
    picked events. Pick times are based on the lag-times found at the maximum
    correlation, providing that correlation is above the min_cc.

    :type detections: list
    :param detections:
        List of :class:`eqcorrscan.core.match_filter.DETECTION` objects.
    :type detect_data: obspy.core.stream.Stream
    :param detect_data:
        All the data needed to cut from - can be a gappy Stream.
    :type template_names: list
    :param template_names:
        List of the template names, used to help identify families of events.
        Must be in the same order as templates.
    :type templates: list
    :param templates:
        List of the templates, templates should be
        :class:`obspy.core.stream.Stream` objects.
    :type shift_len: float
    :param shift_len:
        Shift length allowed for the pick in seconds, will be plus/minus this
        amount - default=0.2
    :type min_cc: float
    :param min_cc:
        Minimum cross-correlation value to be considered a pick, default=0.4.
    :type cores: int
    :param cores:
        Number of cores to use in parallel processing, defaults to one.
    :type interpolate: bool
    :param interpolate:
        Interpolate the correlation function to achieve sub-sample precision.
    :type plot: bool
    :param plot:
        To generate a plot for every detection or not, defaults to False
    :type parallel: bool
    :param parallel: Turn parallel processing on or off.
    :type debug: int
    :param debug: Debug output level, 0-5 with 5 being the most output.
    .

    :returns:
        Catalog of events with picks.  No origin information is included.
        These events can then be written out via
        :func:`obspy.core.event.Catalog.write`, or to Nordic Sfiles using
        :func:`eqcorrscan.utils.sfile_util.eventtosfile` and located
        externally.
    :rtype: obspy.core.event.Catalog

    .. note:: Picks output in catalog are generated relative to the template \
        start-time.  For example, if you generated your template with a \
        pre_pick time of 0.2 seconds, you should expect picks generated by \
        lag_calc to occur 0.2 seconds before the true phase-pick.  This \
        is because we do not currently store template meta-data alongside the \
        templates.

    .. warning:: Because of the above note, origin times will be consistently \
        shifted by the static pre_pick applied to the templates.

    .. note::
        Individual channel cross-correlations are stored as a
        :class:`obspy.core.event.Comment` for each pick, and the summed
        cross-correlation value resulting from these is stored as a
        :class:`obspy.core.event.Comment` in the main
        :class:`obspy.core.event.Event` object.

    .. note::
        The order of events is preserved (e.g. detections[n] == output[n]),
        providing picks have been made for that event.  If no picks have
        been made for an event, it will not be included in the output.
        However, as each detection has an ID associated with it, these can
        be mapped to the output resource_id for each Event in the output
        Catalog. e.g.

            detections[n].id == output[m].resource_id

        if the output[m] is for the same event as detections[n].
    """
    if debug > 0:
        log.setLevel(logging.WARNING)
        ch.setLevel(logging.WARNING)
    if debug > 2:
        log.setLevel(logging.INFO)
        ch.setLevel(logging.INFO)
    if debug > 3:
        log.setLevel(0)
        ch.setLevel(0)
    log.addHandler(ch)
    if debug > 2 and plot:
        prep_plot = True
    else:
        prep_plot = False
    # First check that sample rates are equal for everything
    for tr in detect_data:
        if tr.stats.sampling_rate != detect_data[0].stats.sampling_rate:
            raise LagCalcError('Sampling rates are not equal')
    for template in templates:
        for tr in template:
            if tr.stats.sampling_rate != detect_data[0].stats.sampling_rate:
                raise LagCalcError('Sampling rates are not equal')
    # Work out the delays for each template
    delays = []  # List of tuples of (tempname, (sta, chan, delay))
    zipped_templates = list(zip(template_names, templates))
    detect_stachans = [(tr.stats.station, tr.stats.channel)
                       for tr in detect_data]
    for template in zipped_templates:
        temp_delays = []
        # Remove channels not present in continuous data
        _template = template[1].copy()
        for tr in _template:
            if (tr.stats.station, tr.stats.channel) not in detect_stachans:
                _template.remove(tr)
        for tr in _template:
            temp_delays.append((tr.stats.station, tr.stats.channel,
                                tr.stats.starttime - _template.
                                sort(['starttime'])[0].stats.starttime))
        delays.append((template[0], temp_delays))
        del _template
    # Segregate detections by template, then feed to day_loop
    initial_cat = Catalog()
    for template in zipped_templates:
        log.info('Running lag-calc for template %s' % template[0])
        template_detections = [detection for detection in detections
                               if detection.template_name == template[0]]
        log.info('There are %i detections' % len(template_detections))
        detect_streams = _prepare_data(detect_data=detect_data,
                                       detections=template_detections,
                                       zipped_templates=zipped_templates,
                                       delays=delays, shift_len=shift_len,
                                       plot=prep_plot)
        detect_streams = [detect_stream[1] for detect_stream in detect_streams]
        if len(template_detections) > 0:
            template_cat = _day_loop(detection_streams=detect_streams,
                                     template=template[1], min_cc=min_cc,
                                     detections=template_detections,
                                     interpolate=interpolate, cores=cores,
                                     parallel=parallel)
            initial_cat += template_cat
            if plot:
                for i, event in enumerate(template_cat):
                    if len(event.picks) == 0:
                        log.warning('Made no picks for event at time %s' %
                                    event)
                        continue
                    plot_stream = detect_streams[i].copy()
                    pick_stachans = [(pick.waveform_id.station_code,
                                      pick.waveform_id.channel_code)
                                     for pick in event.picks]
                    for tr in plot_stream:
                        if (tr.stats.station, tr.stats.channel) \
                                not in pick_stachans:
                            plot_stream.remove(tr)
                    template_plot = template[1].copy()
                    for tr in template_plot:
                        if (tr.stats.station, tr.stats.channel) \
                                not in pick_stachans:
                            template_plot.remove(tr)
                    plot_repicked(template=template_plot, picks=event.picks,
                                  det_stream=plot_stream)
    sys.stdout.flush()
    # Order the catalogue to match the input
    output_cat = Catalog()
    for det in detections:
        event = [e for e in initial_cat if e.resource_id == det.id]
        if len(event) == 1:
            output_cat.append(event[0])
        elif len(event) == 0:
            print('No picks made for detection:')
            print(det)
        else:
            raise NotImplementedError('Multiple events with same id,'
                                      ' should not happen')
    return output_cat

Example 115

Project: ISYlib-python
Source File: scapy-watch.py
View license
def pcap_callback(pkt):

    eaddr = None
    ipaddr = None
    pktinfo = None

    t = time.strftime(time_fmt, time.localtime())
    if ARP in pkt and pkt[ARP].op in (1, 2): #who-has or is-at
        eaddr = pkt[ARP].hwsrc
        ipaddr = pkt[ARP].psrc

        try:
            pktinfo = pkt.sprintf("{0}\tARP %ARP.hwsrc% %ARP.psrc% %ARP.op% %ARP.pdst%".format(t))
        except Exception, x:
            print "Scapy_Exception ARP : ", x
            pktinfo = None


# #    elif TCP in pkt:
#        eaddr = pkt[Ether].src
#        pktinfo = pkt.sprintf("{0}\tTCP %Ether.src% %Ether.dst% %IP.src%:%TCP.sport% %IP.dst%:%TCP.dport%".format(t))

#    elif UDP in pkt:
#        eaddr = pkt[Ether].src
#        pktinfo = pkt.sprintf("{0}\t%IP.proto% %Ether.src% %Ether.dst% %IP.src%:%UDP.sport% %IP.dst%:%UDP.dport%".format(t))
#
    elif IP in pkt:
        eaddr = pkt[Ether].src
        ipaddr = pkt[IP].src
        try:
            pktinfo = pkt.sprintf("{0}\tIP %IP.proto% %Ether.src% %Ether.dst% %IP.src% %IP.dst%".format(t))
        except Exception, x:
            print "Scapy_Exception IP : ", x
            pktinfo = None

    elif Ether in pkt:
        eaddr = pkt[Ether].src
        try:
            pktinfo = pkt.sprintf("{0}\tEther %Ether.src% %Ether.dst% ".format(t))
        except Exception, x:
            print "Scapy_Exception Ether : ", x
            pktinfo = None

    elif Dot3 in pkt:
        eaddr = pkt[Dot3].src
        try:
            pktinfo = pkt.sprintf("{0}\tDot3 %Dot3.src% %Dot3.dst% ".format(t))
        except Exception, x:
            print "Scapy_Exception Dot3 : ", x
            print "pkt", pkt, "\n"
            pkt.show()
            pktinfo = None

    else:
        if verbose:
            # print ">> pkt __dict__", pkt.__dict__
            print "pkt", pkt
            print "dir", dir(pkt)
            print "pkt.name", pkt.name
            pkt.show()
            return "???"
        else:
            return None


    if eaddr in mac_targets:
        # print t, "\tmac_{0}".format(eaddr), mac_targets[eaddr].ip
        ti = int(time.time())
        prev_seen = mac_targets[eaddr].last_seen
        time_since = ti - prev_seen

        # update IP address ( if needed )
        #if ipaddr is not None and ipaddr != "0.0.0.0":
        if ipaddr not in [ None, "0.0.0.0", "255.255.255.255" ]:
            if mac_targets[eaddr].ip is None:
                mac_targets[eaddr].ip = ipaddr
                if verbose:
                    print t, "\t A", pkt.summary()
                    # print t, pkt

                    print t, "\tset_ipaddr\t{} to {}\t{}".format(
                        mac_targets[eaddr].mac, mac_targets[eaddr].ip,
                        mac_targets[eaddr].var.name)
            elif mac_targets[eaddr].ip != mac_targets[eaddr].ip:
                if verbose:
                    print t, "\t B", pkt.summary()
                    print t, "\tSet_ipaddr\t{} changed {} -> {}\t{}".format(
                        mac_targets[eaddr].mac, mac_targets[eaddr].ip,
                        ipaddr, mac_targets[eaddr].var.name)
                mac_targets[eaddr].ip = ipaddr


        # dont react to *every* packet in a row
        if (time_since > (time_recheck / 3)) or (mac_targets[eaddr].is_active < 1):
            # print time.strftime(time_fmt, time.localtime()), t, pkt.time, (pkt.time - t)
            if verbose and mac_targets[eaddr].is_active < 1:
                print t, "\t +", pkt.summary()
            mac_targets[eaddr].set_var(1)
        else:
            if verbose > 2:
                print "{}\tpcap_callback: time_since={} > {}".format(t, time_since, (time_recheck / 10))
                print "{}\tpcap_callback: time_since = {} = {} - {}".format(t, time_since,
                    prev_seen, ti)

        mac_targets[eaddr].last_seen = ti
    sys.stdout.flush()

    if verbose > 3:
        return pktinfo
    else:
        return None

Example 116

Project: mock
Source File: util.py
View license
def logOutput(fds, logger, returnOutput=1, start=0, timeout=0, printOutput=False,
              child=None, chrootPath=None, pty=False):
    output = ""
    done = 0

    # set all fds to nonblocking
    for fd in fds:
        flags = fcntl.fcntl(fd, fcntl.F_GETFL)
        if not fd.closed:
            fcntl.fcntl(fd, fcntl.F_SETFL, flags | os.O_NONBLOCK)

    mockbuild_logger = logging.getLogger('mockbuild')
    stored_propagate = mockbuild_logger.propagate
    if printOutput:
        # prevent output being printed twice when log propagates to stdout
        mockbuild_logger.propagate = 0
        sys.stdout.flush()
    try:
        tail = ""
        ansi_escape = re.compile(r'\x1b\[([0-9]{1,2}(;[0-9]{1,2})?)?[m|K]\x0f?')
        while not done:
            if (time.time() - start) > timeout and timeout != 0:
                done = 1
                break

            i_rdy, o_rdy, e_rdy = select.select(fds, [], [], 1)

            if not i_rdy and not o_rdy and not e_rdy:
                if child and child.poll() is not None:
                    logger.info("Child pid '%s' is dead", child.pid)
                    done = True
                    if chrootPath:
                        logger.info("Child dead, killing orphans")
                        orphansKill(chrootPath)
                    continue

            for s in i_rdy:
                # slurp as much input as is ready
                raw = s.read()
                if not raw:
                    done = 1
                    break
                if printOutput:
                    if hasattr(sys.stdout, 'buffer'):
                        # python3 would print binary strings ugly
                        # pylint: disable=no-member
                        sys.stdout.buffer.write(raw)
                    else:
                        print(raw, end='')
                    sys.stdout.flush()
                txt_input = raw.decode(encoding, 'replace')
                lines = txt_input.split("\n")
                if tail:
                    lines[0] = tail + lines[0]
                # we may not have all of the last line
                tail = lines.pop()
                if not lines:
                    continue
                if pty:
                    lines = [process_input(line) for line in lines]
                processed_input = '\n'.join(lines) + '\n'
                if logger is not None:
                    for line in lines:
                        if line != '':
                            line = ansi_escape.sub('', line)
                            logger.debug(line)
                    for h in logger.handlers:
                        h.flush()
                if returnOutput:
                    output += processed_input

        if tail:
            if pty:
                tail = process_input(tail) + '\n'
            if logger is not None:
                logger.debug(tail)
            if returnOutput:
                output += tail
    finally:
        mockbuild_logger.propagate = stored_propagate

    return output

Example 117

Project: cronq
Source File: runner.py
View license
def create_runner(channel):  # noqa

    def run_something(msg):
        tag = msg.delivery_info['delivery_tag']

        make_directory(LOG_PATH)

        def ack():
            channel.basic.ack(tag)

        def reject(requeue=True):
            channel.basic.reject(tag, requeue=requeue)

        def publish_result(body):
            headers = {
                'x-send-datetime': str(datetime.datetime.utcnow()),
                'x-host': str(socket.getfqdn()),
            }
            body.update(headers)
            msg = Message(json.dumps(body))
            channel.basic.publish(msg, 'cronq', 'cronq_results')

        def valid_job(data):
            valid = True
            for key in ['cmd', 'job_id', 'run_id']:
                if not data.get(key, None):
                    logger.debug('Missing {0}'.format(key))
                    valid = False
            return valid

        data = json.loads(str(msg.body))
        if not valid_job(data):
            return reject(requeue=False)

        cmd = data.get('cmd')
        logger.info('[cronq_job_id:{0}] [cronq_run_id:{1}] Starting {2}'.format(
            data.get('job_id'), data.get('run_id'), cmd
        ))
        publish_result({
            'job_id': data.get('job_id'),
            'run_id': data.get('run_id'),
            'start_time': str(datetime.datetime.utcnow()),
            'type': 'starting',
        })
        start = time.time()
        process = None
        try:
            process = subprocess.Popen(
                cmd,
                shell=True,
                stdout=subprocess.PIPE,
                stderr=subprocess.STDOUT,
            )
        except OSError:
            logger.exception('[cronq_job_id:{0}] [cronq_run_id:{1}] Failed job'.format(
                data.get('job_id'), data.get('run_id')
            ))
            end = time.time()
            publish_result({
                'job_id': data.get('job_id'),
                'run_id': data.get('run_id'),
                'run_time': end - start,
                'type': 'failed',
            })
            return reject(requeue=False)

        fd = process.stdout.fileno()
        fl = fcntl.fcntl(fd, fcntl.F_GETFL)
        fcntl.fcntl(fd, fcntl.F_SETFL, fl | os.O_NONBLOCK)

        logger.info('[cronq_job_id:{0}] [cronq_run_id:{1}] Waiting'.format(
            data.get('job_id'), data.get('run_id')
        ))

        splits = FILENAME_REGEX.split(data.get('name', 'UNKNOWN'))
        if len(splits) > 1:
            logfile = '-'.join(splits)
        filename = '{0}/{1}.log'.format(LOG_PATH, logfile.strip('-'))

        handler = logging.handlers.WatchedFileHandler(filename)
        log_to_stdout = bool(os.getenv('CRONQ_RUNNER_LOG_TO_STDOUT', False))

        while True:
            try:
                nextline = process.stdout.readline()
            except IOError:
                nextline = ''

            if nextline == '' and process.poll() is not None:
                break

            if nextline == '':
                time.sleep(0.1 * random.random())
                continue

            try:
                message = nextline.rstrip()
                message = unicodedammit(message)
            except:
                continue

            if message:
                for m in message.splitlines():
                    log_record = logging.makeLogRecord({
                        'msg': message,
                    })
                    handler.emit(log_record)
                    if log_to_stdout:
                        logger.info(u'[cronq_job_id:{0}] [cronq_run_id:{1}] {2}'.format(
                            data.get('job_id'), data.get('run_id'), log_record.getMessage()
                        ))

            time.sleep(0.00001)
            sys.stdout.flush()

        handler.close()

        end = time.time()
        publish_result({
            'job_id': data.get('job_id'),
            'run_id': data.get('run_id'),
            'return_code': process.returncode,
            'run_time': end - start,
            'type': 'finished',
        })
        logger.info('[cronq_job_id:{0}] [cronq_run_id:{1}] [cronq_exit_code:{2}] Done'.format(
            data.get('job_id'), data.get('run_id'), process.returncode
        ))
        ack()

    return run_something

Example 118

Project: Wallace
Source File: test_models.py
View license
    def test_models(self):

        """####################
        #### Test Network ####
        ####################"""

        print("")
        print("Testing models: Network", end="\r")
        sys.stdout.flush()

        # create test network:
        net = models.Network()
        self.db.add(net)
        self.db.commit()
        net = models.Network.query.one()

        # create a participant
        participant = models.Participant(worker_id=str(1), hit_id=str(1), assignment_id=str(1), mode="test")
        self.db.add(participant)
        self.db.commit()

        # create some nodes
        node = models.Node(network=net)
        agent = Agent(network=net, participant=participant)
        source = Source(network=net)

        # create vectors
        source.connect(direction="to", whom=agent)
        agent.connect(direction="both", whom=node)

        # create some infos
        info = models.Info(origin=agent, contents="ethwth")
        gene = Gene(origin=source, contents="hkhkhkh")

        # conditionally transmit and transform
        source.transmit(what=models.Info)
        agent.receive()
        agent.transmit(what=Gene)
        models.Transformation(info_in=gene, info_out=info)

        # Test attributes

        assert net.id == 1
        assert isinstance(net.creation_time, datetime)
        assert net.property1 is None
        assert net.property2 is None
        assert net.property3 is None
        assert net.property4 is None
        assert net.property5 is None
        assert net.failed is False
        assert net.time_of_death is None
        assert net.type == "network"
        assert isinstance(net.max_size, int)
        assert net.max_size == 1e6
        assert isinstance(net.full, bool)
        assert net.full is False
        assert isinstance(net.role, unicode)
        assert net.role == "default"

        # test __repr__()
        assert repr(net) == "<Network-1-network with 3 nodes, 3 vectors, 2 infos, 1 transmissions and 1 transformations>"

        # test __json__()
        assert net.__json__() == {
            "id": 1,
            "type": "network",
            "max_size": 1e6,
            "full": False,
            "role": "default",
            "creation_time": net.creation_time,
            "failed": False,
            "time_of_death": None,
            "property1": None,
            "property2": None,
            "property3": None,
            "property4": None,
            "property5": None
        }

        # test nodes()
        for n in [node, agent, source]:
            assert n in net.nodes()

        assert net.nodes(type=Agent) == [agent]

        assert net.nodes(failed=True) == []
        for n in [node, agent, source]:
            assert n in net.nodes(failed="all")

        assert net.nodes(participant_id=1) == [agent]

        # test size()
        assert net.size() == 3
        assert net.size(type=Source) == 1
        assert net.size(type=Agent) == 1
        assert net.size(failed=True) == 0
        assert net.size(failed="all") == 3

        # test infos()
        assert len(net.infos(failed="all")) == 2
        assert len(net.infos(type=models.Info, failed="all")) == 2
        assert len(net.infos(type=Gene, failed="all")) == 1
        assert len(net.infos(type=Gene)) == 1
        assert len(net.infos(failed=True)) == 0

        # test Network.transmissions()
        assert len(net.transmissions(failed="all")) == 1
        assert len(net.transmissions(failed=True)) == 0
        assert len(net.transmissions(failed=False)) == 1
        assert len(net.transmissions(status="pending", failed="all")) == 0
        assert len(net.transmissions(status="received", failed="all")) == 1

        # test Network.transformations()
        assert len(net.transformations(failed="all")) == 1
        assert len(net.transformations(failed="all", type=Mutation)) == 0
        assert len(net.transformations(failed="all", type=models.Transformation)) == 1

        for t in net.transformations(failed="all"):
            assert type(t.node) == Agent

        # test latest_transmission_recipient
        assert net.latest_transmission_recipient() == agent

        # test Network.vectors()
        assert len(net.vectors(failed="all")) == 3
        assert len(net.vectors(failed=False)) == 3
        assert len(net.vectors(failed=True)) == 0

        # test fail()
        net.fail()
        assert net.nodes() == []
        assert len(net.nodes(failed=True)) == 3
        assert len(net.nodes(failed="all")) == 3
        assert net.infos() == []
        assert net.transmissions() == []
        assert net.vectors() == []
        assert net.transformations() == []

        print("Testing models: Network    passed!")
        sys.stdout.flush()

Example 119

Project: ISYlib-python
Source File: scapy-watch.py
View license
def ping_loop():
    """
        init stage:
            loops though mac_targets and try to arp ping each one

        loop stage:
            sleep for a while

            loop though mac_targets and try to arp ping each one we have not seen in a while

            check for timeout on non-responsive targets and set their state to 0

    """

    if verbose:
        print time.strftime(time_fmt, time.localtime()), "\tping_loop init", current_thread().name

    for c in mac_targets.values():

        if c.ip is None:
            #icmp_a, icmp_u = icmp_ping("255.255.255.255", c.mac)
            continue

        icmp_a = None

        # icmp_a, icmp_u = icmp_ping(c.ip)
        arp_a, _ = arp_ping(c.ip)
        # arping(c.ip)

        if arp_a or icmp_a:
            c.set_var(1)
            c.last_seen = int(time.time())
            # print time.strftime(time_fmt, time.localtime()), "PRE",
            # arp_a.summary()

    if verbose:
        print time.strftime(time_fmt, time.localtime()), "\tping_loop start"

    while True:


        if verbose:
            print time.strftime(time_fmt, time.localtime()), "\tping_loop sleep start"

        sys.stdout.flush()
        time.sleep(time_sleep)

        if verbose:
            print time.strftime(time_fmt, time.localtime()), "\tping_loop sleep complete"

        for c in mac_targets.values():

            time_now = float(time.time()) # int(time.time())
            time_since = time_now - c.last_seen

            strtm = time.strftime(time_fmt, time.localtime())

            if time_since >= time_recheck:
                if verbose & 0x02:
                    print strtm, "\tping_loop: {} time_since >= time_recheck".format(c.mac), c.var.name
                    # print "arp_pinging"

                if c.ip is not None:
                    a, u = arp_ping(c.ip)

                    #if len(a) < 1:
                    #    a, u = icmp_ping(c.ip, c.mac)

                    if len(a):
                        if verbose: # or (delta and c.var.value < 1):
                            print strtm, "\tseen", c.mac, c.var.name, \
                                "time_since = {:3.2f} sec = {}".format(
                                    time_since,
                                    format_sec(time_since))
                            print strtm, "\t",
                            a.summary()
                            # last_seen, time.strftime(time_fmt, time.localtime(c.last_seen)), c.var.name
                        c.set_var(1)
                        c.last_seen = int(time.time())
                        continue

                # wait to see if sniffer sees it
                # time.sleep(5)
                # strtm = time.strftime(time_fmt, time.localtime())
                # time_since = int(time.time()) - c.last_seen

            if time_since >= time_away:
                if verbose and c.ip is None:
                    #print "{}\tping_loop: time_since >= time_away, last_seen = {}".format(
                    #    strtm,
                    #    time.strftime(time_fmt, time.localtime(c.last_seen)))
                    print "\t", c.mac, c.ip, c.var.name

                if delta and c.is_active == 1:
                    print "{}\t{}  ping_loop: time_since = {:3.2f} sec = {}".format(
                        strtm,
                        c.mac,
                        time_since,
                        format_sec(time_since))

                # set inital last_seen to start file of prog
                if c.is_active == -1:
                    c.last_seen = int(start_time)

                # dont set var for *every* timeout in a row, unless needed
                if c.is_active or (time_now - c.set_var_time) > (time_away * 2):
                    c.set_var(0)
                else:
                    if verbose:
                        print "\tpass", c.var.name, c.var.value

Example 120

Project: ISYlib-python
Source File: _isy_printevent.py
View license
def _print_event(*arg):

    ddat = arg[0]
    # mydat = arg[1]
    exml = arg[2]

# Event Dat:
# {'control': 'DOF', 'node': '16 6C D2 7', 'eventInfo': None, 'Event-seqnum': '141', 'action': '0', 'Event-sid': 'uuid:40'}
# <?xml version="1.0"?><Event seqnum="141" sid="uuid:40"><control>DOF</control><action>0</action><node>16 6C D2 7</node><eventInfo></eventInfo></Event>
#
    ti = time.strftime('%X')
    try:
        if "control" not in ddat or  ddat["control"] is None:
            return

        control_val =  ddat["control"]

        if "action" in ddat and ddat["action"] is not None:
            action_val =  _action_val(ddat["action"])
        else :
            action_val =  None

        if control_val in EVENT_CTRL_ACTION and action_val:
            action_str = EVENT_CTRL_ACTION[control_val].get(action_val, action_val)
        else :
            action_str = ""

        node = ddat.get("node", "")
        if node is None:
            node = ""

        control_str = EVENT_CTRL.get(control_val, control_val)

        evi = ""

        if ddat["control"] in ["_0"]:
            pass

        elif ddat["control"] == "ERR":
            print "{!s:<7} {!s:<4}\t{!s:<12}{!s:<12}\t{!s}".format(
                    ti, ddat['Event-seqnum'],
                    "ERR",
                    ddat['node'], action_str)

            return


        #elif ddat["control"] in ["DOF", "DON", "BMAN", "SMAN", "FDUP", "FDSTOP", "FDDOWN" ]:
        #    print "{!s:<7} {!s:<4}\t{!s:<12}{!s}\t{!s}".format(
        #            ti, ddat['Event-seqnum'], node, control_str, action_str)
        #    return

        elif ddat["control"] in ["ST", "RR", "OL", "DOF", "DON", "DFOF", "DFON",
                                "BMAN", "SMAN", "FDUP", "FDSTOP", "FDDOWN" ]:

            if ddat["eventInfo"] is not None:
                evi = ddat["eventInfo"]
            else :
                evi = ""
            print "{!s:<7} {!s:<4}\t{!s:<12}{!s:<12}\t{!s:<12}\t{!s}".format(
                ti, ddat["Event-seqnum"], node, control_str, action_val, evi)
            return

        elif ddat["control"] == "_1":
            # 'on': None, 'f': '140630 20:55:55', 's': '31', 'r': '140630 20:55:55', 'nr': None, 'id': '1E'}

            #action = EVENT_CTRL_ACTION[ddat["control"]].get(ddat['action'], ddat['action'])
            status = ""

            if action_val == '0':
                st=[ ]

                if 'id' in ddat["eventInfo"]:
                    st.append("id={}".format(ddat["eventInfo"]['id']))

                if 's' in ddat["eventInfo"]:
                    st.append("status={}".format(ddat["eventInfo"]['s']))

                if 'on' in ddat["eventInfo"]:
                    st.append( "enabled=true")
                if 'off' in ddat["eventInfo"]:
                    st.append( "enabled=false")

                if 'rr' in ddat["eventInfo"]:
                    st.append( "runAtStartup=true")
                if 'nr' in ddat["eventInfo"]:
                    st.append( "runAtStartup=false")

                if 'r' in ddat["eventInfo"]:
                    st.append("lastRunTime={}".format(ddat["eventInfo"]['r']))

                if 'f' in ddat["eventInfo"]:
                    st.append("lastFinishTime={}".format(ddat["eventInfo"]['f']))

                if 'nsr' in ddat["eventInfo"]:
                    st.append("nextScheduledRunTime={}".format(
                                ddat["eventInfo"]['nsr']))

                status = " ".join(st)

            elif action_val == '6':
                status = "{!s:<12} {!s}:{!s} {!s} {!s}".format(
                    ddat['node'],
                    ddat['eventInfo']['var']['var-type'],
                    ddat['eventInfo']['var']['var-id'],
                    ddat['eventInfo']['var']['val'],
                    ddat['eventInfo']['var']['ts'])

            elif action_val == '7':
                status = "{!s:<12} {!s}:{!s} {!s}".format(
                    ddat['node'],
                    ddat['eventInfo']['var']['var-type'],
                    ddat['eventInfo']['var']['var-id'],
                    ddat['eventInfo']['var']['init'])

            else:
                if isinstance( ddat['eventInfo'], dict):
                    status = " ".join(["{}={}".format(a,b) for a, b in ddat['eventInfo'].items()] )
                elif ddat['eventInfo'] is None:
                    status=""
                else:
                    status = ddat['eventInfo']

            print "{!s:<7} {!s:<4}\t{!s:<12}{!s}\t{!s}\t{!s}".format(
                ti, ddat["Event-seqnum"], node, control_str, action_str, status)

            return

        elif ddat["control"] in [ "_3", "_4", "_5", "_6", "_7", "_8", "_9",
                        "_10", "_11", "_12", "_13", "_14", "_15", "_16", "_19",
                        "_20", "_21", "_22" ] :
            d = ddat['eventInfo']
            if isinstance( d, dict):
                status = " ".join(["{}={}".format(a,b) for a, b in d.items()] )
            elif d is None:
                status=""
            else:
                status = eventInfo

            #status = ddat['eventInfo']
            print "{!s:<7} {!s:<4}\t{!s:<12}{!s:<12}\t{!s:<12}\t{!s}".format(
                ti, ddat["Event-seqnum"], node, control_str, action_str, status)

            return


#        elif ddat["control"] == "_11":
#            status = ddat['eventInfo']
#            status="value={} unit={}".format(
#                        ddat['eventInfo'].get('value', ""),
#                        ddat['eventInfo'].get('unit', ""))
#
#            print "{!s:<7} {!s:<4}\t{!s:<12}\t{!s}\t{!s}\t{!s}".format(
#                ti, ddat["Event-seqnum"], node, control_str, action_str, status)
#            return


        elif ddat["control"] == "_17":
            if action_val == '1':
                status = "total={!s:<12} lastReadTS={!s}".format(
                        ddat['eventInfo'].get('total', ""),
                        ddat['eventInfo'].get('lastReadTS', "")
                    )
            else:
                status = ddat.get('eventInfo', "")

            print "{!s:<7} {!s:<4}\t{!s:<12}{!s:<12}\t{!s:<12}\t{!s}".format(
                ti, ddat["Event-seqnum"], node, control_str, action_str, status)
            return

        elif ddat["control"] == "_18":
            if 'ZBNetwork' in ddat['eventInfo']:
                d = ddat['eventInfo']['ZBNetwork']
                status = " ".join(["{}={}".format(a,b) for a, b in d.items()] )
            else:
                status = ddat['eventInfo']
            print "{!s:<7} {!s:<4}\t{!s:<12}{!s:<12}\t{!s:<12}\t{!s}".format(
                ti, ddat["Event-seqnum"], node, control_str, action_str, status)
            return

        elif ddat["control"] == "_23":
            if 'PortalStatus' in ddat['eventInfo']:
                d = ddat['eventInfo']['PortalStatus']
                status = " ".join(["{}={}".format(a,b) for a, b in d.items()] )
            else:
                status = ddat['eventInfo']
            print "{!s:<7} {!s:<4}\t{!s:<12}{!s:<12}\t{!s:<12}\t{!s}".format(
                ti, ddat["Event-seqnum"], node, control_str, action_str, status)
            return

        else:
            status = ddat.get('eventInfo', "")
            print "{!s:<7} {!s:<4}\t{!s:<12}{!s:<12}\t{!s:<12}\t{!s}".format(
                ti, ddat["Event-seqnum"], node, control_str, action_str, status)
#           return

#           if node is None:
#               node = ""
#           if action is None:
#               action = ddat.get('action', "")
#           print "{!s:<7} {!s:<4}\t{} : {} : {}\t{!s:<12}".format(
#               ti, ddat['Event-seqnum'],
#               node,
#               control_str,
#               action,
#               ddat.get('eventInfo', "-") )

            print "Event Dat : \n\t", ddat, "\n\t", exml


        sys.stdout.flush()
        #print ddat
        # print data
    except Exception as e:
        print("Unexpected error:", sys.exc_info()[0])
        print(e)
        print(ddat)
        raise
        # print data
    finally:
        pass

Example 121

Project: awslogs
Source File: core.py
View license
    def list_logs(self):
        streams = []
        if self.log_stream_name != self.ALL_WILDCARD:
            streams = list(self._get_streams_from_pattern(self.log_group_name, self.log_stream_name))
            if len(streams) > self.FILTER_LOG_EVENTS_STREAMS_LIMIT:
                raise exceptions.TooManyStreamsFilteredError(
                     self.log_stream_name,
                     len(streams),
                     self.FILTER_LOG_EVENTS_STREAMS_LIMIT
                )
            if len(streams) == 0:
                raise exceptions.NoStreamsFilteredError(self.log_stream_name)

        max_stream_length = max([len(s) for s in streams]) if streams else 10
        group_length = len(self.log_group_name)

        # Note: filter_log_events paginator is broken
        # ! Error during pagination: The same next token was received twice
        do_wait = object()

        def generator():
            """Yield events into trying to deduplicate them using a lru queue.
            AWS API stands for the interleaved parameter that:
                interleaved (boolean) -- If provided, the API will make a best
                effort to provide responses that contain events from multiple
                log streams within the log group interleaved in a single
                response. That makes some responses return some subsequent
                response duplicate events. In a similar way when awslogs is
                called with --watch option, we need to findout which events we
                have alredy put in the queue in order to not do it several
                times while waiting for new ones and reusing the same
                next_token. The site of this queue is MAX_EVENTS_PER_CALL in
                order to not exhaust the memory.
            """
            interleaving_sanity = deque(maxlen=self.MAX_EVENTS_PER_CALL)
            kwargs = {'logGroupName': self.log_group_name,
                      'interleaved': True}

            if streams:
                kwargs['logStreamNames'] = streams

            if self.start:
                kwargs['startTime'] = self.start

            if self.end:
                kwargs['endTime'] = self.end

            if self.filter_pattern:
                kwargs['filterPattern'] = self.filter_pattern

            while True:
                response = self.client.filter_log_events(**kwargs)

                for event in response.get('events', []):
                    if event['eventId'] not in interleaving_sanity:
                        interleaving_sanity.append(event['eventId'])
                        yield event

                if 'nextToken' in response:
                    kwargs['nextToken'] = response['nextToken']
                else:
                    yield do_wait

        def consumer():
            for event in generator():

                if event is do_wait:
                    if self.watch:
                        time.sleep(1)
                        continue
                    else:
                        return

                output = []
                if self.output_group_enabled:
                    output.append(
                        self.color(
                            self.log_group_name.ljust(group_length, ' '),
                            'green'
                        )
                    )
                if self.output_stream_enabled:
                    output.append(
                        self.color(
                            event['logStreamName'].ljust(max_stream_length,
                                                         ' '),
                            'cyan'
                        )
                    )
                if self.output_timestamp_enabled:
                    output.append(
                        self.color(
                            milis2iso(event['timestamp']),
                            'yellow'
                        )
                    )
                if self.output_ingestion_time_enabled:
                    output.append(
                        self.color(
                            milis2iso(event['ingestionTime']),
                            'blue'
                        )
                    )

                message = event['message']
                if self.query is not None and message[0] == '{':
                    parsed = json.loads(event['message'])
                    message = self.query_expression.search(parsed)
                    if not isinstance(message, six.string_types):
                        message = json.dumps(message)
                output.append(message)

                print(' '.join(output))
                sys.stdout.flush()
        try:
            consumer()
        except KeyboardInterrupt:
            print('Closing...\n')
            os._exit(0)

Example 122

Project: CCLib
Source File: cc2510.py
View license
	def writeCODE(self, offset, data, erase=False, verify=False, showProgress=False):
		"""
		Fully automated function for writing the Flash memory.

		WARNING: This requires DMA operations to be unpaused ( use: self.pauseDMA(False) )
		"""

		# Prepare DMA-0 for DEBUG -> RAM (using DBG_BW trigger)
		self.configDMAChannel( 0, 0x6260, 0x0000, 0x1F, tlen=self.bulkBlockSize, srcInc=0, dstInc=1, priority=1, interrupt=True )
		# Prepare DMA-1 for RAM -> FLASH (using the FLASH trigger)
		self.configDMAChannel( 1, 0x0000, 0x6273, 0x12, tlen=self.bulkBlockSize, srcInc=1, dstInc=0, priority=2, interrupt=True )

		# Reset flags
		self.clearFlashStatus()
		self.clearDMAIRQ(0)
		self.clearDMAIRQ(1)
		self.disarmDMAChannel(0)
		self.disarmDMAChannel(1)
		flashRetries = 0

		# Split in 2048-byte chunks
		iOfs = 0
		while (iOfs < len(data)):

			# Check if we should show progress
			if showProgress:
				print "\r    Progress %0.0f%%... " % (iOfs*100/len(data)),
				sys.stdout.flush()

			# Get next page
			iLen = min( len(data) - iOfs, self.bulkBlockSize )

			# Update DMA configuration if we have less than bulk-block size data 
			if (iLen < self.bulkBlockSize):
				self.configDMAChannel( 0, 0x6260, 0x0000, 0x1F, tlen=iLen, srcInc=0, dstInc=1, priority=1, interrupt=True )
				self.configDMAChannel( 1, 0x0000, 0x6273, 0x12, tlen=iLen, srcInc=1, dstInc=0, priority=2, interrupt=True )

			# Upload to RAM through DMA-0
			self.armDMAChannel(0)
			self.brustWrite( data[iOfs:iOfs+iLen] )

			# Wait until DMA-0 raises interrupt
			while not self.isDMAIRQ(0):
				time.sleep(0.010)

			# Clear DMA IRQ flag
			self.clearDMAIRQ(0)

			# Calculate the page where this data belong to
			fAddr = offset + iOfs
			fPage = int( fAddr / self.flashPageSize )

			# Calculate FLASH address High/Low bytes
			# for writing (addressable as 32-bit words)
			fWordOffset = int(fAddr / 4)
			cHigh = (fWordOffset >> 8) & 0xFF
			cLow = fWordOffset & 0xFF
			self.writeXDATA( 0x6271, [cLow, cHigh] )

			# Debug
			#print "[@%04x: p=%i, ofs=%04x, %02x:%02x]" % (fAddr, fPage, fWordOffset, cHigh, cLow),
			#sys.stdout.flush()

			# Check if we should erase page first
			if erase:
				# Select the page to erase using FADDRH[7:1]
				#
				# NOTE: Specific to (CC2530, CC2531, CC2540, and CC2541),
				#       the CC2533 uses FADDRH[6:0]
				#
				cHigh = (fPage << 1)
				cLow = 0
				self.writeXDATA( 0x6271, [cLow, cHigh] )
				# Set the erase bit
				self.setFlashErase()
				# Wait until flash is not busy any more
				while self.isFlashBusy():
					time.sleep(0.010)

			# Upload to FLASH through DMA-1
			self.armDMAChannel(1)
			self.setFlashWrite()

			# Wait until DMA-1 raises interrupt
			while not self.isDMAIRQ(1):
				# Also check for errors
				if self.isFlashAbort():
					self.disarmDMAChannel(1)
					raise IOError("Flash page 0x%02x is locked!" % fPage)
				time.sleep(0.010)

			# Clear DMA IRQ flag
			self.clearDMAIRQ(1)

			# Check if we should verify
			if verify:
				verifyBytes = self.readCODE(fAddr, iLen)
				for i in range(0, iLen):
					if verifyBytes[i] != data[iOfs+i]:
						if flashRetries < 3:
							print "\n[Flash Error at @0x%04x, will retry]" % (fAddr+i)
							flashRetries += 1
							continue
						else:
							raise IOError("Flash verification error on offset 0x%04x" % (fAddr+i))
			flashRetries = 0

			# Forward to next page
			iOfs += iLen

		if showProgress:
			print "\r    Progress 100%... OK"

Example 123

Project: androguard
Source File: androdd.py
View license
def export_apps_to_format(filename,
                          s,
                          output,
                          methods_filter=None,
                          jar=None,
                          decompiler_type=None,
                          format=None):
    print "Dump information %s in %s" % (filename, output)

    if not os.path.exists(output):
        print "Create directory %s" % output
        os.makedirs(output)
    else:
        print "Clean directory %s" % output
        androconf.rrmdir(output)
        os.makedirs(output)

    methods_filter_expr = None
    if methods_filter:
        methods_filter_expr = re.compile(methods_filter)

    output_name = output
    if output_name[-1] != "/":
        output_name = output_name + "/"

    dump_classes = []
    for _, vm, vmx in s.get_objects_dex():
        print "Decompilation ...",
        sys.stdout.flush()

        if decompiler_type == "dex2jad":
            vm.set_decompiler(decompiler.DecompilerDex2Jad(
                vm, androconf.CONF["PATH_DEX2JAR"], androconf.CONF["BIN_DEX2JAR"
                              ], androconf.CONF["PATH_JAD"],
                androconf.CONF["BIN_JAD"], androconf.CONF["TMP_DIRECTORY"]))
        elif decompiler_type == "dex2winejad":
            vm.set_decompiler(decompiler.DecompilerDex2WineJad(
                vm, androconf.CONF["PATH_DEX2JAR"], androconf.CONF["BIN_DEX2JAR"
                              ], androconf.CONF["PATH_JAD"],
                androconf.CONF["BIN_WINEJAD"], androconf.CONF["TMP_DIRECTORY"]))
        elif decompiler_type == "ded":
            vm.set_decompiler(decompiler.DecompilerDed(
                vm, androconf.CONF["PATH_DED"], androconf.CONF["BIN_DED"],
                androconf.CONF["TMP_DIRECTORY"]))
        elif decompiler_type == "dex2fernflower":
            vm.set_decompiler(decompiler.DecompilerDex2Fernflower(
                vm, androconf.CONF["PATH_DEX2JAR"], androconf.CONF[
                    "BIN_DEX2JAR"
                ], androconf.CONF["PATH_FERNFLOWER"], androconf.CONF[
                    "BIN_FERNFLOWER"
                ], androconf.CONF["OPTIONS_FERNFLOWER"
                                 ], androconf.CONF["TMP_DIRECTORY"]))

        print "End"

        if options.jar:
            print "jar ...",
            filenamejar = decompiler.Dex2Jar(
                vm, androconf.CONF["PATH_DEX2JAR"],
                androconf.CONF["BIN_DEX2JAR"],
                androconf.CONF["TMP_DIRECTORY"]).get_jar()
            shutil.move(filenamejar, output + "classes.jar")
            print "End"

        for method in vm.get_methods():
            if methods_filter_expr:
                msig = "%s%s%s" % (method.get_class_name(), method.get_name(),
                                   method.get_descriptor())
                if not methods_filter_expr.search(msig):
                    continue

            filename_class = valid_class_name(method.get_class_name())
            create_directory(filename_class, output)

            print "Dump %s %s %s ..." % (method.get_class_name(),
                                         method.get_name(),
                                         method.get_descriptor()),

            filename_class = output_name + filename_class
            if filename_class[-1] != "/":
                filename_class = filename_class + "/"

            descriptor = method.get_descriptor()
            descriptor = descriptor.replace(";", "")
            descriptor = descriptor.replace(" ", "")
            descriptor = descriptor.replace("(", "-")
            descriptor = descriptor.replace(")", "-")
            descriptor = descriptor.replace("/", "_")

            filename = filename_class + method.get_name() + descriptor
            if len(method.get_name() + descriptor) > 250:
                all_identical_name_methods = vm.get_methods_descriptor(
                    method.get_class_name(), method.get_name())
                pos = 0
                for i in all_identical_name_methods:
                    if i.get_descriptor() == method.get_descriptor():
                        break
                    pos += 1

                filename = filename_class + method.get_name() + "_%d" % pos

            buff = method2dot(vmx.get_method(method))

            if format:
                print "%s ..." % format,
                method2format(filename + "." + format, format, None, buff)

            if method.get_class_name() not in dump_classes:
                print "source codes ...",
                current_class = vm.get_class(method.get_class_name())
                current_filename_class = valid_class_name(
                    current_class.get_name())

                current_filename_class = output_name + current_filename_class + ".java"
                with open(current_filename_class, "w") as fd:
                    fd.write(current_class.get_source())
                dump_classes.append(method.get_class_name())

            print "bytecodes ...",
            bytecode_buff = dvm.get_bytecodes_method(vm, vmx, method)
            with open(filename + ".ag", "w") as fd:
                fd.write(bytecode_buff)
            print

Example 124

Project: multiprocess
Source File: ex_pool.py
View license
def test():
    print 'cpuCount() = %d\n' % cpuCount()
    
    #
    # Create pool
    #
    
    PROCESSES = 4
    print 'Creating pool with %d processes\n' % PROCESSES
    pool = Pool(PROCESSES)    

    #
    # Tests
    #

    TASKS = [(mul, (i, 7)) for i in range(10)] + \
            [(plus, (i, 8)) for i in range(10)]

    results = [pool.applyAsync(calculate, t) for t in TASKS]
    imap_it = pool.imap(calculatestar, TASKS)
    imap_unordered_it = pool.imapUnordered(calculatestar, TASKS)

    print 'Ordered results using pool.applyAsync():'
    for r in results:
        print '\t', r.get()
    print

    print 'Ordered results using pool.imap():'        
    for x in imap_it:
        print '\t', x
    print

    print 'Unordered results using pool.imapUnordered():'
    for x in imap_unordered_it:
        print '\t', x
    print

    print 'Ordered results using pool.map() --- will block till complete:'
    for x in pool.map(calculatestar, TASKS):
        print '\t', x
    print    

    #
    # Simple benchmarks
    #

    N = 100000
    print 'def pow3(x): return x**3'
    
    t = time.time()
    A = map(pow3, xrange(N))
    print '\tmap(pow3, xrange(%d)):\n\t\t%s seconds' % \
          (N, time.time() - t)
    
    t = time.time()
    B = pool.map(pow3, xrange(N))
    print '\tpool.map(pow3, xrange(%d)):\n\t\t%s seconds' % \
          (N, time.time() - t)

    t = time.time()
    C = list(pool.imap(pow3, xrange(N), chunksize=N//8))
    print '\tlist(pool.imap(pow3, xrange(%d), chunksize=%d)):\n\t\t%s' \
          ' seconds' % (N, N//8, time.time() - t)
    
    assert A == B == C, (len(A), len(B), len(C))
    print
    
    L = [None] * 1000000
    print 'def noop(x): pass'
    print 'L = [None] * 1000000'
    
    t = time.time()
    A = map(noop, L)
    print '\tmap(noop, L):\n\t\t%s seconds' % \
          (time.time() - t)
    
    t = time.time()
    B = pool.map(noop, L)
    print '\tpool.map(noop, L):\n\t\t%s seconds' % \
          (time.time() - t)

    t = time.time()
    C = list(pool.imap(noop, L, chunksize=len(L)//8))
    print '\tlist(pool.imap(noop, L, chunksize=%d)):\n\t\t%s seconds' % \
          (len(L)//8, time.time() - t)

    assert A == B == C, (len(A), len(B), len(C))
    print    

    del A, B, C, L

    #
    # Test error handling
    #

    print 'Testing error handling:'

    try:
        print pool.apply(f, (5,))
    except ZeroDivisionError:
        print '\tGot ZeroDivisionError as expected from pool.apply()'
    else:
        raise AssertionError, 'expected ZeroDivisionError'

    try:
        print pool.map(f, range(10))
    except ZeroDivisionError:
        print '\tGot ZeroDivisionError as expected from pool.map()'
    else:
        raise AssertionError, 'expected ZeroDivisionError'
            
    try:
        print list(pool.imap(f, range(10)))
    except ZeroDivisionError:
        print '\tGot ZeroDivisionError as expected from list(pool.imap())'
    else:
        raise AssertionError, 'expected ZeroDivisionError'

    it = pool.imap(f, range(10))
    for i in range(10):
        try:
            x = it.next()
        except ZeroDivisionError:
            if i == 5:
                pass
        except StopIteration:
            break
        else:
            if i == 5:
                raise AssertionError, 'expected ZeroDivisionError'
            
    assert i == 9
    print '\tGot ZeroDivisionError as expected from IMapIterator.next()'
    print
    
    #
    # Testing timeouts
    #
    
    print 'Testing ApplyResult.get() with timeout:',
    res = pool.applyAsync(calculate, TASKS[0])
    while 1:
        sys.stdout.flush()
        try:
            sys.stdout.write('\n\t%s' % res.get(0.02))
            break
        except TimeoutError:
            sys.stdout.write('.')
    print
    print

    print 'Testing IMapIterator.next() with timeout:',
    it = pool.imap(calculatestar, TASKS)
    while 1:
        sys.stdout.flush()
        try:
            sys.stdout.write('\n\t%s' % it.next(0.02))
        except StopIteration:
            break
        except TimeoutError:
            sys.stdout.write('.')
    print
    print
            
    #
    # Testing callback
    #

    print 'Testing callback:'
    
    A = []
    B = [56, 0, 1, 8, 27, 64, 125, 216, 343, 512, 729]
        
    r = pool.applyAsync(mul, (7, 8), callback=A.append)
    r.wait()

    r = pool.mapAsync(pow3, range(10), callback=A.extend)
    r.wait()

    if A == B:
        print '\tcallbacks succeeded\n'
    else:
        print '\t*** callbacks failed\n\t\t%s != %s\n' % (A, B)
    
    #
    # Check there are no outstanding tasks
    #
    
    assert not pool._cache, 'cache = %r' % pool._cache

    #
    # Check close() methods
    #

    print 'Testing close():'

    for worker in pool._pool:
        assert worker.isAlive()

    result = pool.applyAsync(time.sleep, [0.5])
    pool.close()
    pool.join()

    assert result.get() is None

    for worker in pool._pool:
        assert not worker.isAlive()

    print '\tclose() succeeded\n'

    #
    # Check terminate() method
    #

    print 'Testing terminate():'

    pool = Pool(2)
    ignore = pool.apply(pow3, [2])
    results = [pool.applyAsync(time.sleep, [10]) for i in range(10)]
    pool.terminate()
    pool.join()

    for worker in pool._pool:
        assert not worker.isAlive()

    print '\tterminate() succeeded\n'

    #
    # Check garbage collection
    #

    print 'Testing garbage collection:'

    pool = Pool(2)
    processes = pool._pool
    
    ignore = pool.apply(pow3, [2])
    results = [pool.applyAsync(time.sleep, [10]) for i in range(10)]

    del results, pool

    time.sleep(0.2)
    
    for worker in processes:
        assert not worker.isAlive()

    print '\tgarbage collection succeeded\n'

Example 125

Project: exabgp
Source File: healthcheck.py
View license
def loop(options):
    """Main loop."""
    states = enum(
        "INIT",                 # Initial state
        "DISABLED",             # Disabled state
        "RISING",               # Checks are currently succeeding.
        "FALLING",              # Checks are currently failing.
        "UP",                   # Service is considered as up.
        "DOWN",                 # Service is considered as down.
    )

    def exabgp(target):
        """Communicate new state to ExaBGP"""
        if target not in (states.UP, states.DOWN, states.DISABLED):
            return
        logger.info("send announces for %s state to ExaBGP", target)
        metric = vars(options).get("{0}_metric".format(str(target).lower()))
        for ip in options.ips:
            if options.withdraw_on_down:
                command = "announce" if target is states.UP else "withdraw"
            else:
                command = "announce"
            announce = "route {0} next-hop {1}".format(
                str(ip),
                options.next_hop or "self")

            if command == "announce":
                announce = "{0} med {1}".format(announce, metric)
                if options.community:
                    announce = "{0} community [ {1} ]".format(
                        announce,
                        options.community)
                if options.large_community:
                    announce = "{0} large-community [ {1} ]".format(
                        announce,
                        options.large_community)
                if options.as_path:
                    announce = "{0} as-path [ {1} ]".format(
                        announce,
                        options.as_path)
            logger.debug("exabgp: %s %s", command, announce)
            print("{0} {1}".format(command, announce))
            metric += options.increase
        sys.stdout.flush()

    def trigger(target):
        """Trigger a state change and execute the appropriate commands"""
        # Shortcut for RISING->UP and FALLING->UP
        if target == states.RISING and options.rise <= 1:
            target = states.UP
        elif target == states.FALLING and options.fall <= 1:
            target = states.DOWN

        # Log and execute commands
        logger.debug("Transition to %s", str(target))
        cmds = []
        cmds.extend(vars(options).get("{0}_execute".format(
            str(target).lower()), []) or [])
        cmds.extend(vars(options).get("execute", []) or [])
        for cmd in cmds:
            logger.debug("Transition to %s, execute `%s`",
                         str(target), cmd)
            env = os.environ.copy()
            env.update({"STATE": str(target)})
            with open(os.devnull, "w") as fnull:
                subprocess.call(
                    cmd, shell=True, stdout=fnull, stderr=fnull, env=env)

        return target

    def one(checks, state):
        """Execute one loop iteration."""
        disabled = (options.disable is not None and
                    os.path.exists(options.disable))
        successful = disabled or check(options.command, options.timeout)
        # FSM
        if state != states.DISABLED and disabled:
            state = trigger(states.DISABLED)
        elif state == states.INIT:
            if successful and options.rise <= 1:
                state = trigger(states.UP)
            elif successful:
                state = trigger(states.RISING)
                checks = 1
            else:
                state = trigger(states.FALLING)
                checks = 1
        elif state == states.DISABLED:
            if not disabled:
                state = trigger(states.INIT)
        elif state == states.RISING:
            if successful:
                checks += 1
                if checks >= options.rise:
                    state = trigger(states.UP)
            else:
                state = trigger(states.FALLING)
                checks = 1
        elif state == states.FALLING:
            if not successful:
                checks += 1
                if checks >= options.fall:
                    state = trigger(states.DOWN)
            else:
                state = trigger(states.RISING)
                checks = 1
        elif state == states.UP:
            if not successful:
                state = trigger(states.FALLING)
                checks = 1
        elif state == states.DOWN:
            if successful:
                state = trigger(states.RISING)
                checks = 1
        else:
            raise ValueError("Unhandled state: {0}".format(str(state)))

        # Send announces. We announce them on a regular basis in case
        # we lose connection with a peer.
        exabgp(state)
        return checks, state

    checks = 0
    state = states.INIT
    while True:
        checks, state = one(checks, state)

        # How much we should sleep?
        if state in (states.FALLING, states.RISING):
            time.sleep(options.fast)
        else:
            time.sleep(options.interval)

Example 126

Project: calibre
Source File: simple.py
View license
    def process_links(self, soup, baseurl, recursion_level, into_dir='links'):
        res = ''
        diskpath = os.path.join(self.current_dir, into_dir)
        if not os.path.exists(diskpath):
            os.mkdir(diskpath)
        prev_dir = self.current_dir
        try:
            self.current_dir = diskpath
            tags = list(soup.findAll('a', href=True))

            for c, tag in enumerate(tags):
                if self.show_progress:
                    print '.',
                    sys.stdout.flush()
                sys.stdout.flush()
                iurl = self.absurl(baseurl, tag, 'href', filter=recursion_level != 0)
                if not iurl:
                    continue
                nurl = self.normurl(iurl)
                if self.filemap.has_key(nurl):  # noqa
                    self.localize_link(tag, 'href', self.filemap[nurl])
                    continue
                if self.files > self.max_files:
                    return res
                linkdir = 'link'+str(c) if into_dir else ''
                linkdiskpath = os.path.join(diskpath, linkdir)
                if not os.path.exists(linkdiskpath):
                    os.mkdir(linkdiskpath)
                try:
                    self.current_dir = linkdiskpath
                    dsrc = self.fetch_url(iurl)
                    newbaseurl = dsrc.newurl
                    if len(dsrc) == 0 or \
                       len(re.compile('<!--.*?-->', re.DOTALL).sub('', dsrc).strip()) == 0:
                        raise ValueError('No content at URL %r'%iurl)
                    if callable(self.encoding):
                        dsrc = self.encoding(dsrc)
                    elif self.encoding is not None:
                        dsrc = dsrc.decode(self.encoding, 'replace')
                    else:
                        dsrc = xml_to_unicode(dsrc, self.verbose)[0]

                    st = time.time()
                    soup = self.get_soup(dsrc, url=iurl)
                    self.log.debug('Parsed %s in %.1f seconds' % (iurl, time.time() - st))

                    base = soup.find('base', href=True)
                    if base is not None:
                        newbaseurl = base['href']
                    self.log.debug('Processing images...')
                    self.process_images(soup, newbaseurl)
                    if self.download_stylesheets:
                        self.process_stylesheets(soup, newbaseurl)

                    _fname = basename(iurl)
                    if not isinstance(_fname, unicode):
                        _fname.decode('latin1', 'replace')
                    _fname = _fname.encode('ascii', 'replace').replace('%', '').replace(os.sep, '')
                    _fname = ascii_filename(_fname)
                    _fname = os.path.splitext(_fname)[0][:120] + '.xhtml'
                    res = os.path.join(linkdiskpath, _fname)
                    self.downloaded_paths.append(res)
                    self.filemap[nurl] = res
                    if recursion_level < self.max_recursions:
                        self.log.debug('Processing links...')
                        self.process_links(soup, newbaseurl, recursion_level+1)
                    else:
                        self.process_return_links(soup, newbaseurl)
                        self.log.debug('Recursion limit reached. Skipping links in', iurl)

                    if newbaseurl and not newbaseurl.startswith('/'):
                        for atag in soup.findAll('a', href=lambda x: x and x.startswith('/')):
                            atag['href'] = urlparse.urljoin(newbaseurl, atag['href'], True)
                    if callable(self.postprocess_html_ext):
                        soup = self.postprocess_html_ext(soup,
                                c==0 and recursion_level==0 and not getattr(self, 'called_first', False),
                                self.job_info)

                        if c==0 and recursion_level == 0:
                            self.called_first = True

                    save_soup(soup, res)
                    self.localize_link(tag, 'href', res)
                except Exception as err:
                    if isinstance(err, AbortArticle):
                        raise
                    self.failed_links.append((iurl, traceback.format_exc()))
                    self.log.exception('Could not fetch link', iurl)
                finally:
                    self.current_dir = diskpath
                    self.files += 1
        finally:
            self.current_dir = prev_dir
        if self.show_progress:
            print
        return res

Example 127

Project: pywikibot-core
Source File: script_tests.py
View license
    def __new__(cls, name, bases, dct):
        """Create the new class."""
        def test_execution(script_name, args=[]):
            is_autorun = '-help' not in args and script_name in auto_run_script_list

            def test_skip_script(self):
                raise unittest.SkipTest(
                    'Skipping execution of auto-run scripts (set '
                    'PYWIKIBOT2_TEST_AUTORUN=1 to enable) "{0}"'.format(script_name))

            def testScript(self):
                cmd = [script_name]

                if args:
                    cmd += args

                data_in = script_input.get(script_name)

                timeout = 0
                if is_autorun:
                    timeout = 5

                if self._results and script_name in self._results:
                    error = self._results[script_name]
                    if isinstance(error, StringTypes):
                        stdout = None
                    else:
                        stdout, error = error
                else:
                    stdout = None
                    error = None

                test_overrides = {}
                if not hasattr(self, 'net') or not self.net:
                    test_overrides['pywikibot.Site'] = 'None'

                result = execute_pwb(cmd, data_in, timeout=timeout, error=error,
                                     overrides=test_overrides)

                stderr = result['stderr'].splitlines()
                stderr_sleep = [l for l in stderr
                                if l.startswith('Sleeping for ')]
                stderr_other = [l for l in stderr
                                if not l.startswith('Sleeping for ')]
                if stderr_sleep:
                    print(u'\n'.join(stderr_sleep))

                if result['exit_code'] == -9:
                    print(' killed', end='  ')

                if error:
                    self.assertIn(error, result['stderr'])

                    exit_codes = [0, 1, 2, -9]
                elif not is_autorun:
                    if stderr_other == []:
                        stderr_other = None
                    if stderr_other is not None:
                        self.assertIn('Use -help for further information.',
                                      stderr_other)
                        self.assertNotIn('-help', args)
                    else:
                        self.assertIn('Global arguments available for all',
                                      result['stdout'])

                    exit_codes = [0]
                else:
                    # auto-run
                    exit_codes = [0, -9]

                    if (not result['stdout'] and not result['stderr']):
                        print(' auto-run script unresponsive after %d seconds'
                              % timeout, end=' ')
                    elif 'SIMULATION: edit action blocked' in result['stderr']:
                        print(' auto-run script simulated edit blocked',
                              end='  ')
                    else:
                        print(' auto-run script stderr within %d seconds: %r'
                              % (timeout, result['stderr']), end='  ')

                self.assertNotIn('Traceback (most recent call last)',
                                 result['stderr'])
                self.assertNotIn('deprecated', result['stderr'].lower())

                # If stdout doesnt include global help..
                if 'Global arguments available for all' not in result['stdout']:
                    # Specifically look for deprecated
                    self.assertNotIn('deprecated', result['stdout'].lower())
                    if result['stdout'] == '':
                        result['stdout'] = None
                    # But also complain if there is any stdout
                    if stdout is not None and result['stdout'] is not None:
                        self.assertIn(stdout, result['stdout'])
                    else:
                        self.assertIsNone(result['stdout'])

                self.assertIn(result['exit_code'], exit_codes)

                sys.stdout.flush()

            if not enable_autorun_tests and is_autorun:
                return test_skip_script
            return testScript

        argument = '-' + dct['_argument']

        for script_name in script_list:
            # force login to be the first, alphabetically, so the login
            # message does not unexpectedly occur during execution of
            # another script.
            # unrunnable script tests are disabled by default in load_tests()

            if script_name == 'login':
                test_name = 'test__login'
            else:
                test_name = 'test_' + script_name

            cls.add_method(dct, test_name,
                           test_execution(script_name, [argument]),
                           'Test running %s %s.' % (script_name, argument))

            if script_name in dct['_expected_failures']:
                dct[test_name] = unittest.expectedFailure(dct[test_name])
            elif script_name in dct['_allowed_failures']:
                dct[test_name] = allowed_failure(dct[test_name])

            # Disable test by default in nosetests
            if script_name in unrunnable_script_list:
                # flag them as an expectedFailure due to py.test (T135594)
                dct[test_name] = unittest.expectedFailure(dct[test_name])
                dct[test_name].__test__ = False

        return super(TestScriptMeta, cls).__new__(cls, name, bases, dct)

Example 128

Project: Ezhil-Lang
Source File: ezhil_web.py
View license
    def do_ezhil_execute(self,program):
        # execute the Ezhil Interpreter with string @program
        print("<html> <head> <title>Ezhil interpreter</title>")
        print("""<script src="./Blob.js"></script>
<script src="./FileSaver.js"></script>
<script lang="text/javascript">
  function download(filename, content) {
    saveAs( new Blob([content],{type: "application/x-ezhil;charset=utf-8"}), filename);
  }
</script>""")
        print("</head><body> ")
        print("<!-- ") #suppress exec stdout
        close_comment = False #and useful to debug a live site
        program_fmt = """<TABLE>
        <TR><TD>
        <TABLE>
        <TR>
        <TD><font color=\"blue\"><OL>"""
        
        if ( self.debug ):
            print( "Source program <BR />" )
            print("program = ",program,"<BR />")
            print( "*"*60 )
            print("<BR />")
        
        program_fmt += "\n".join(["<li>%s</li>"%(prog_line)  for line_no,prog_line in enumerate(program.split('\n'))])
        program_fmt += """</OL></font></TD></TR>\n</TABLE></TD><TD>"""
        sys.stdout.flush()
        sys.stderr.flush()
        # run the interpreter in a sandbox and capture the output hopefully
        try:
            failed = False
            obj = EzhilFileExecuter( file_input = [program], redirectop = True, TIMEOUT = 60*2 ) # 2 minutes
            obj.run()

            [f1,f2,progout] = obj.get_output()
            for f in [f1,f2]:
                try:
                    os.unlink( f )
                except Exception as e:
                    pass
            
        
            #SUCCESS_STRING = "<H2> Your program executed correctly! Congratulations. </H2>"            
            if ( self.debug ):
                print(obj.exitcode)
                print(progout)
            print("-->")
            close_comment = True
            if obj.exitcode != 0 and EzhilWeb.error_qualifiers(progout):
                if ( self.debug ):
                    print("Exitcode => ",obj.exitcode)
                    print(progout)
                self.img_outcome = "<IMG width='64' SRC='../icons/%s' alt='failure' />"%EzhilWeb.get_image('failure')
                op = "%s <B>%s Failed Execution, with parsing or evaluation error</B> for program with <font color=\"red\">error <pre>%s</pre> </font></TD></TR></TABLE>"%(program_fmt,self.img_outcome,progout)
                failed = True
            else:
                failed = False
                self.img_outcome = "<IMG width='64' SRC='../icons/%s' alt='success' />"%EzhilWeb.get_image('success')
                op = "%s <B>%s Succeeded Execution</B> for program with output, <BR/> <font color=\"green\"><pre>%s</pre></font></TD></TR>"%(program_fmt,self.img_outcome,progout)
                op = op + saveYourCode(program)
                op = op + "</TABLE>"
        except Exception as e:
            raise e
            if( not close_comment ):
                print("-->")
            if ( self.debug ):
                print("FAILED EXECUTION")
                print(str(e))
            failed = True
            self.img_outcome = "<IMG SRC='../icons/%s' width='64' alt='failure' />"%EzhilWeb.get_image('failure')
            op = "%s <B>%s FAILED Execution</B> for program with <font color=\"red\">error <pre>%s</pre> </font></TD></TR>"%(program_fmt,self.img_outcome,str(e)) 
            op = op + saveYourCode(program)
            op = op + "</TABLE>"
        if ( self.debug ):
            print("Output file")
            print(obj.get_output())
        prev_page = "<script>\ndocument.write(\"Navigate back to your source program : <a href='#' onClick='history.back();return false;'>Go Back</a>\");\n</script><BR />\n<HR/>\n"
        print(prev_page)
        #op = self.img_outcome + op        
        if failed:
            op = "<H2> Your program has some errors! Try correcting it and re-evaluate the code</H2><HR/><BR/>"+op
        else:
            op = "<H2> Your program executed correctly! Congratulations. </H2><HR/><BR/>"+op
        print(op.decode("utf-8"))
        print("</body></html>\n")
 
        return op

Example 129

Project: GBVideoPlayer
Source File: convert.py
View license
def convert(folder, n_frames):
    palette = ["\xFF\xFF"] * 32
    frames = []
    out = open("%s/video.gbv" % (folder,) , "w")
    skipped = 0
    current_skip = 0
    rows = {}
    for i in xrange(0, n_frames):
        print "\r %d/%d (%d skipped)" % (i + 1, n_frames, skipped),
        sys.stdout.flush()

        image = png.Reader("%s/%d.png" % (folder, i)).asRGB8()[2]
        offset = 0
        colors = []
        frame = []

        location = out.tell()

        # Create palette, attempt to keep previous colors to improve row-back-
        # referencing compression rate.
        for y, line in enumerate(image):
            for x in xrange(len(line) / 3):
                pixel = rgb2gb(*line[x*3: x*3+3])
                if pixel not in colors:
                    colors += [pixel]

        palette = [(color if color in colors else "\xFF\xFF") for color in palette]
        for color in colors:
            if color not in palette:
                palette[palette.index("\xFF\xFF")] = color

        binary_palette = (''.join(palette))[::-1]
        assert len(palette) == PALETTE_COLORS

        ## Write the frame header

        # Write the repeat count, it will be increased later if needed
        out.write("\x01")
        # Write the palette
        out.write(binary_palette)

        # Fill the bank if we're in the red zone after the header
        fill_bank(out)

        # Write the data
        image = png.Reader("%s/%d.png" % (folder, i)).asRGB8()[2]
        diff = 0xffffffff

        for y, line in enumerate(image):
            row = ""
            # Write a single row
            for x in xrange(len(line) / 3):
                rgb = line[x*3: x*3+3]

                pixel = rgb2gb(*rgb)
                frame += [pixel]
                # This byte is the Y scroll value required to display request
                # color in this specific row. This value must be odd - it is
                # required to support row-back-referencing. This calculation
                # can be improved to enhance RBR compression rate.
                offset = chr(((palette.index(pixel) * 4 - (y%144) + 256 + 1) | 1) % 256)
                row += offset

            if True: # row not in rows or i == 0: # Row compression has some
                     # unfixed bugs, so it's disabled for now; the compression
                     # rate isn't that great anyway. I probably something silly
                     # I did broke it, but I can't remember if it ever worked.
                rows[row] = out.tell()
                out.write(row)
            else:
                row_location = rows[row]
                bank = row_location / BANK_SIZE + 1
                address = row_location % BANK_SIZE + 0x4000
                out.write(chr(bank / 256 * 2))
                out.write(chr(bank % 256))
                out.write(chr(address % 256))
                out.write(chr(address / 256))

            # Fill the bank if we're in the red zone after a row
            fill_bank(out)

        if len(frames):
            # Calculate diff between this and the previous frame
            diff = frame_diff(frame, frames[-1][1], 256)

        frames += [(location, frame)]

        if (diff < SKIP_THRESHOLD and current_skip < 127):
            # Update the repeat count in the previous frame
            current_skip += 1
            out.seek(frames[-2][0])
            out.write(chr(current_skip * 2 + 1))

            # Delete the frame we just wrote
            out.seek(frames[-1][0])
            out.truncate()

            # Delete rows used in this frame from the rows dict
            bad_keys = []
            for key, value in rows.iteritems():
                if value > frames[-1][0]:
                    bad_keys += [key]

            for key in bad_keys:
                rows.pop(key)

            # Remove this frame from the frame list
            frames = frames[:-1]

            # For statistics
            skipped += 1
        else:
            # Reset the skip - we actually wrote our frame
            current_skip = 0

Example 130

Project: writemdict
Source File: testSalsa20.py
View license
def test( module, module_name ):
    print("===== Testing", module_name, "version", module._version, "=====")
    from sys import stdout

    passed = True

    if 1:  # Test these if the module has them:
        if "rot32" in module.__dict__:
            passed &= test_rot32( module.rot32, module_name+".rot32" )
            # Compare to slow version:
            passed &= test_rot32( rot32long, "rot32long" )
        print()

        if "add32" in module.__dict__:
            passed &= test_add32( module.add32, module_name+".add32" )

    if 1 and passed:
        test_salsa20core( module, module_name )

    if 1 and passed:
        rounds  = 8                     # may be 8, 12, or 20
    
        if 0:
            message = loadfmfile('testdata.txt')
        else:
            message = b'Kilroy was here!  ...there, and everywhere.'
        key     = b'myKey67890123456'    # 16 or 32 bytes, exactly
        nonce   = b'aNonce'              # do better in real life
        IV      = (nonce+b'*'*8)[:8]     # force to exactly 64 bits

        print("Testing decrypt(encrypt(short_message))==short_message...")
        # encrypt
        s20 = salsa20_test_classes[module_name]( key, IV, rounds )
        ciphertxt = s20.encryptBytes(message)
        
        # decrypt
        s20 = salsa20_test_classes[module_name]( key, IV, rounds )
        plaintxt  = s20.encryptBytes(ciphertxt)
    
        if message == plaintxt:
            print('    *** good ***')
        else:
            print('    *** bad ***')
            passed = False


    if 1 and passed:    # one known good 8-round test vector 
        print("Testing known 64-byte message and key...")
        rounds = 8      # must be 8 for this test
        message = b'\x00'*64
        key = binascii.unhexlify('00000000000000000000000000000002')
        IV  = binascii.unhexlify('0000000000000000')
        out64 = bytestring("""
               06C80B8CEC60F0C2E73EB6ED5DCB1B9C
               39B210F1AB76FEDF1A6B7AE370DA0F20
               0CEBCAD6EF6E57AC80E4375C035FA44D
               3AE4DC2C2507757DAF37B14F36643489""")
        s20 = salsa20_test_classes[module_name]( key, IV, rounds )
        s20.setIV(IV)
        ciphertxt = s20.encryptBytes(message)
        s20 = salsa20_test_classes[module_name]( key, IV, rounds )
        plaintxt  = s20.encryptBytes(ciphertxt)
        if (message == plaintxt and
            ciphertxt[:64] == out64):
            print('    *** vector 1 good ***')
        else:
            print('    *** vector 1 bad ***')
            passed = False
        
    if 1 and passed:    # one known good 8-round test vector 
        print("Testing known key and 64k message...")
        rounds = 8      # must be 8 for this test
        message = b'\x00'*65536
        key = binascii.unhexlify('0053A6F94C9FF24598EB3E91E4378ADD')
        IV  = binascii.unhexlify('0D74DB42A91077DE')
        out64 = bytestring("""
               75FCAE3A3961BDC7D2513662C24ADECE
               995545599FF129006E7A6EE57B7F33A2
               6D1B27C51EA15E8F956693472DC23132
               FCD90FB0E352D26AF4DCE5427193CA26""")
        out65536 = bytestring("""
               EA75A566C431A10CED804CCD45172AD1
               EC4930E9869372B8EDDF303098A8910C
               EE123BF849C51A33554BA1445E6B6268
               4921F36B77EADC9681A2BB9DDFEC2FC8""")
        s20 = salsa20_test_classes[module_name]( key, IV, rounds )
        ciphertxt = s20.encryptBytes(message)
        s20 = salsa20_test_classes[module_name]( key, IV, rounds )
        plaintxt  = s20.encryptBytes(ciphertxt)
        if (message == plaintxt and
            ciphertxt[:64] == out64 and
            ciphertxt[65472:] == out65536):
            print('    *** vector 2 good ***')
        else:
            print('    *** vector 2 bad ***')
            passed = False

    if 1 and passed:    # some rough speed tests
        from time import time
        from math import ceil

        print("Speed tests...")
        names = {}
        speeds = {}
        message_lens = [ 64, 2**16 ]
        #                    64-byte message     65536-byte message
        # Salsa20/4000: 12345678.9 bytes/sec   12345678.9 bytes/sec
        namefmt = "%13s"
        print(namefmt % " ", end=" ")
        msg_len_fmt =  "%7d-byte message  "
        speed_fmt =    "%10.1f bytes/sec  "
        for msg_len in message_lens:
            print(msg_len_fmt % msg_len,end=" ")
        print()

        for nRounds in [ 8, 20, 4000 ]:
            names[ nRounds ] = "Salsa20/" + repr(nRounds) + ":"
            print(namefmt % names[ nRounds ], end=" ")
            speeds[ nRounds ] = {}
            if nRounds <= 20: lens = message_lens
            else:             lens = message_lens[ 0 : -1 ]
            for msg_len in lens:
                message = b'\x00' * msg_len
                key = binascii.unhexlify('00000000000000000000000000000002')
                IV  = binascii.unhexlify('0000000000000000')
                s20 = salsa20_test_classes[module_name]( key, IV, 20 )
                s20.force_nRounds( nRounds )
                nreps = 1
                duration = 4.0
                while duration < 5: # sec.
                    # Aim for 6 seconds:
                    nreps = int( ceil( nreps * min( 4, 6.0/duration ) ) )
                    start = time()
                    for i in range( nreps ):
                        ciphertxt = s20.encryptBytes(message)
                    duration = time() - start
                speeds[ nRounds ][ msg_len ] = msg_len * nreps / duration
                print(speed_fmt % speeds[ nRounds ][ msg_len ], end=" ")
                stdout.flush()
            print()

    return passed

Example 131

Project: pythia
Source File: run.py
View license
def run_test_on(filename):
    """run one test and returns the number of errors"""
    if not show_details:
        f = open(filename)
        comment = f.readline().strip(" \n\"'")
        f.close()
        print(table_header % (filename[2:-3], comment), end='')
    sum_errors = {}

    if filename.endswith('.html'):
        run_html_test( filename, sum_errors )
        return sum_errors

    def display(function):
        global _test_description
        _test_description = function.__doc__
        if show_details:
            print('\n<%s>\n' % function.__doc__)

        errors = function(filename)
        if errors:
            if not show_details:
                print(table_cell % ''.join('%s%d' % (k[0], v)
                                            for k, v in errors.items()),
                      end='')
        else:
            if not show_details:
                print(table_cell % 'OK', end='')
        sys.stdout.flush()

        for k, v in errors.items():
            sum_errors[k] = sum_errors.get(k, 0) + v

        if show_details:
            print('-'*77)

    if 'requirejs' not in filename and not filename.startswith('./go/'):
        display(run_python_test_on)
        display(run_python3_test_on)
        if pypy_runnable:
            display(run_pypy_test_on)
        if old_pypy_runnable:
            display(run_old_pypy_test_on)

    global js
    do_js_test = not filename.startswith( ('./go/', './asm/', './rust/', './c++') )
    if do_js_test:
        js = translate_js(
            filename,
            javascript=True,
            multioutput=filename.startswith('./threads/' or filename.startswith('./bench/webworker'))
        )
        if rhino_runnable:
            display(run_pythonjs_test_on)
        if node_runnable:
            display(run_pythonjs_test_on_node)

        if nodewebkit_runnable:
            display(run_pythonjs_test_on_nodewebkit)

    ## TODO more optimized js settings pythonjs-minimal.js ##
    #if '--no-javascript-mode' not in sys.argv and do_js_test:
    #    js = translate_js(filename, javascript=True, multioutput=filename.startswith('./threads/' or filename.startswith('./bench/webworker')))
    #    if rhino_runnable:
    #        display(run_pythonjsjs_test_on)
    #    if node_runnable:
    #        display(run_pythonjsjs_test_on_node)
    #    if nodewebkit_runnable:
    #        display(run_pythonjsjs_test_on_nodewebkit)


    if 'requirejs' not in filename:

        if dart_runnable:
            js = translate_js(filename, javascript=False, dart=True)
            display(run_pythonjs_dart_test_on_dart)

        if dart2js_runnable and node_runnable:
            js = translate_js(filename, javascript=False, dart=True)
            display(run_pythonjs_dart_test_on_node)

        if coffee_runnable and node_runnable:
            js = translate_js(filename, javascript=False, dart=False, coffee=True)
            display(run_pythonjs_coffee_test_on_node)

        if luajs_runnable and node_runnable:
            js = translate_js(filename, luajs=True)
            display(run_pythonjs_luajs_test_on_node)

        if lua_runnable:
            js = translate_js(filename, lua=True)
            display(run_pythonjs_lua_test_on_lua)

        if luajit_runnable:
            js = translate_js(filename, lua=True)
            display(run_pythonjs_lua_test_on_luajit)

        if go_runnable:
            js = translate_js(filename, go=True)
            display(run_pythonjs_go_test)

        if gopherjs_runnable:
            js = translate_js(filename, gopherjs=True)
            display(run_pythonjs_gopherjs_test)

        if rust_runnable:
            js = translate_js(filename, rust=True)
            display(run_pythonjs_rust_test)

        if cpp_runnable:
            js = translate_js(filename, cpp=True)
            display(run_pythonjs_cpp_test)


    print()
    return sum_errors

Example 132

Project: multiprocess
Source File: ex_pool.py
View license
def test():
    print 'cpuCount() = %d\n' % cpuCount()
    
    #
    # Create pool
    #
    
    PROCESSES = 4
    print 'Creating pool with %d processes\n' % PROCESSES
    pool = Pool(PROCESSES)    

    #
    # Tests
    #

    TASKS = [(mul, (i, 7)) for i in range(10)] + \
            [(plus, (i, 8)) for i in range(10)]

    results = [pool.apply_async(calculate, t) for t in TASKS]
    imap_it = pool.imap(calculatestar, TASKS)
    imap_unordered_it = pool.imap_unordered(calculatestar, TASKS)

    print 'Ordered results using pool.apply_async():'
    for r in results:
        print '\t', r.get()
    print

    print 'Ordered results using pool.imap():'        
    for x in imap_it:
        print '\t', x
    print

    print 'Unordered results using pool.imap_unordered():'
    for x in imap_unordered_it:
        print '\t', x
    print

    print 'Ordered results using pool.map() --- will block till complete:'
    for x in pool.map(calculatestar, TASKS):
        print '\t', x
    print    

    #
    # Simple benchmarks
    #

    N = 100000
    print 'def pow3(x): return x**3'
    
    t = time.time()
    A = map(pow3, xrange(N))
    print '\tmap(pow3, xrange(%d)):\n\t\t%s seconds' % \
          (N, time.time() - t)
    
    t = time.time()
    B = pool.map(pow3, xrange(N))
    print '\tpool.map(pow3, xrange(%d)):\n\t\t%s seconds' % \
          (N, time.time() - t)

    t = time.time()
    C = list(pool.imap(pow3, xrange(N), chunksize=N//8))
    print '\tlist(pool.imap(pow3, xrange(%d), chunksize=%d)):\n\t\t%s' \
          ' seconds' % (N, N//8, time.time() - t)
    
    assert A == B == C, (len(A), len(B), len(C))
    print
    
    L = [None] * 1000000
    print 'def noop(x): pass'
    print 'L = [None] * 1000000'
    
    t = time.time()
    A = map(noop, L)
    print '\tmap(noop, L):\n\t\t%s seconds' % \
          (time.time() - t)
    
    t = time.time()
    B = pool.map(noop, L)
    print '\tpool.map(noop, L):\n\t\t%s seconds' % \
          (time.time() - t)

    t = time.time()
    C = list(pool.imap(noop, L, chunksize=len(L)//8))
    print '\tlist(pool.imap(noop, L, chunksize=%d)):\n\t\t%s seconds' % \
          (len(L)//8, time.time() - t)

    assert A == B == C, (len(A), len(B), len(C))
    print    

    del A, B, C, L

    #
    # Test error handling
    #

    print 'Testing error handling:'

    try:
        print pool.apply(f, (5,))
    except ZeroDivisionError:
        print '\tGot ZeroDivisionError as expected from pool.apply()'
    else:
        raise AssertionError, 'expected ZeroDivisionError'

    try:
        print pool.map(f, range(10))
    except ZeroDivisionError:
        print '\tGot ZeroDivisionError as expected from pool.map()'
    else:
        raise AssertionError, 'expected ZeroDivisionError'
            
    try:
        print list(pool.imap(f, range(10)))
    except ZeroDivisionError:
        print '\tGot ZeroDivisionError as expected from list(pool.imap())'
    else:
        raise AssertionError, 'expected ZeroDivisionError'

    it = pool.imap(f, range(10))
    for i in range(10):
        try:
            x = it.next()
        except ZeroDivisionError:
            if i == 5:
                pass
        except StopIteration:
            break
        else:
            if i == 5:
                raise AssertionError, 'expected ZeroDivisionError'
            
    assert i == 9
    print '\tGot ZeroDivisionError as expected from IMapIterator.next()'
    print
    
    #
    # Testing timeouts
    #
    
    print 'Testing ApplyResult.get() with timeout:',
    res = pool.apply_async(calculate, TASKS[0])
    while 1:
        sys.stdout.flush()
        try:
            sys.stdout.write('\n\t%s' % res.get(0.02))
            break
        except TimeoutError:
            sys.stdout.write('.')
    print
    print

    print 'Testing IMapIterator.next() with timeout:',
    it = pool.imap(calculatestar, TASKS)
    while 1:
        sys.stdout.flush()
        try:
            sys.stdout.write('\n\t%s' % it.next(0.02))
        except StopIteration:
            break
        except TimeoutError:
            sys.stdout.write('.')
    print
    print
            
    #
    # Testing callback
    #

    print 'Testing callback:'
    
    A = []
    B = [56, 0, 1, 8, 27, 64, 125, 216, 343, 512, 729]
        
    r = pool.apply_async(mul, (7, 8), callback=A.append)
    r.wait()

    r = pool.map_async(pow3, range(10), callback=A.extend)
    r.wait()

    if A == B:
        print '\tcallbacks succeeded\n'
    else:
        print '\t*** callbacks failed\n\t\t%s != %s\n' % (A, B)
    
    #
    # Check there are no outstanding tasks
    #
    
    assert not pool._cache, 'cache = %r' % pool._cache

    #
    # Check close() methods
    #

    print 'Testing close():'

    for worker in pool._pool:
        assert worker.is_alive()

    result = pool.apply_async(time.sleep, [0.5])
    pool.close()
    pool.join()

    assert result.get() is None

    for worker in pool._pool:
        assert not worker.is_alive()

    print '\tclose() succeeded\n'

    #
    # Check terminate() method
    #

    print 'Testing terminate():'

    pool = Pool(2)
    ignore = pool.apply(pow3, [2])
    results = [pool.apply_async(time.sleep, [10]) for i in range(10)]
    pool.terminate()
    pool.join()

    for worker in pool._pool:
        assert not worker.is_alive()

    print '\tterminate() succeeded\n'

    #
    # Check garbage collection
    #

    print 'Testing garbage collection:'

    pool = Pool(2)
    processes = pool._pool
    
    ignore = pool.apply(pow3, [2])
    results = [pool.apply_async(time.sleep, [10]) for i in range(10)]

    del results, pool

    time.sleep(0.2)
    
    for worker in processes:
        assert not worker.is_alive()

    print '\tgarbage collection succeeded\n'

Example 133

Project: pyspaces
Source File: cloning.py
View license
    def __init__(self, process_obj):
        """Execute linux clone.

        Create a child process in new namespace(s);
        allow UID and GID mappings to be specified when
        creating a user namespace.

        Raises:
          OSError: can not execute glibc.clone function

        """
        sys.stdout.flush()
        sys.stderr.flush()
        self.process_obj = process_obj
        self.returncode = None
        self.pipe_fd = os.pipe()

        # clone attributes
        flags = process_obj.__dict__.get('clone_flags', 0)
        uid_map = process_obj.__dict__.get('uid_map', "")
        gid_map = process_obj.__dict__.get('gid_map', "")
        map_zero = process_obj.__dict__.get('map_zero', False)
        proc = process_obj.__dict__.get('proc', '/proc')

        # Create the child in new namespace(s)
        child = CFUNCTYPE(c_int)(self.child)
        child_stack = create_string_buffer(STACK_SIZE)
        child_stack_pointer = c_void_p(cast(child_stack, c_void_p).value + STACK_SIZE)

        self.pid = libc.clone(child, child_stack_pointer, flags | SIGCHLD)

        if self.pid == -1:
            e = get_errno()
            raise OSError(e, os.strerror(e))

        # Update the UID and GID maps in the child
        def arg2map(arg):
            #    int: map given uid to root
            #    str: like int or in format
            #    ' '.join((<start uid in new ns>,
            #              <start uid in current ns>,
            #              <range to mapping>
            #    )). Example "0 1000 1" will map 1000 uid as root,
            #    "0 1000 1,1 1001 1" will map also 1001 as uid 1.
            #    list: list of int or str
            if type(arg) is int:
                return "0 %d 1" % arg
            if not hasattr(arg, '__iter__'):
                arg = arg.split(',')
            if ' ' not in arg[0]:
                arg = ['%d %s 1' % (i, d) for i, d in enumerate(arg)]
            return '\n'.join(arg)

        if uid_map or map_zero:
            map_path = "%s/%s/uid_map" % (proc, self.pid)
            if map_zero or type(uid_map) is bool:
                uid_map = "0 %d 1" % os.getuid()
            else:
                uid_map = arg2map(uid_map)
            self.update_map(uid_map, map_path)

        if gid_map or map_zero:
            # Writing "deny" to the /proc/[pid]/setgroups file before writing to
            # /proc/[pid]/gid_map will permanently disable setgroups(2) in a user
            # namespace and allow writing to /proc/[pid]/gid_map without having the
            # CAP_SETGID capability in the parent user namespace.
            with open("%s/%s/setgroups" % (proc, self.pid), 'w') as f:
                f.write("deny")
            map_path = "%s/%s/gid_map" % (proc, self.pid)
            if map_zero or type(gid_map) is bool:
                gid_map = "0 %d 1" % os.getgid()
            else:
                gid_map = arg2map(gid_map)
            self.update_map(gid_map, map_path)

        # Close the write end of the pipe, to signal to the child that we
        # have updated the UID and GID maps
        os.close(self.pipe_fd[1])
        self.sentinel = self.pipe_fd[0]

Example 134

Project: pyeq2
Source File: generateOutput.py
View license
def PoorlyDefined_A():
    specificBaseName = inspect.stack()[0][3]
    htmlTitle = 'Data With A Poorly Defined Region'
    htmlText = '''
This can be mitigated by taking additional<br>
data in the region that is poorly defined.'''
    thumbnailAnimationName = specificBaseName + '_small.gif'
    fullsizeAnimationName = specificBaseName + '_large.gif'
    
    print specificBaseName

    Ymax = 1000.0
    Ymin = -1000.0
    
    equation = pyeq2.Models_2D.Polynomial.Linear('SSQABS')
    dimensionality = equation.GetDimensionality()
    equation.dataCache.DependentDataContainsZeroFlag = True # do not need relative error calculations, this flag turns them off
        
    xbase = numpy.array([0.0, 60.0, 120.0])
    equation.dataCache.allDataCacheDictionary['Weights'] = []
    
    #numpy.random.seed(3)
    #randomArray = (numpy.random.random_sample(len(xbase)+1) - 0.5) * 3.0
    randomArray = numpy.array([-50.0, 50.0, 51.0, -50.0])

    for i in range(0, 360):
        extraZerosString = ''
        if i < 100:
            extraZerosString = '0'
        if i < 10:
            extraZerosString = '00'
        
        numpy.random.seed(3)
        if not i % largeAnimationModulus:
            print i,;sys.stdout.flush()
            
            equation = pyeq2.Models_2D.Polynomial.Quadratic('SSQABS')
            xdata = xbase  * (0.45 * numpy.sin(numpy.radians(i)) + 0.5)
            xdata = numpy.append(xdata, numpy.array([120.0]))
            equation.dataCache.allDataCacheDictionary['IndependentData'] = numpy.array([xdata, numpy.ones(len(xdata))])
            IndependentDataArray = equation.dataCache.allDataCacheDictionary['IndependentData']
            equation.dataCache.allDataCacheDictionary['DependentData'] =  xdata + randomArray
            DependentDataArray = equation.dataCache.allDataCacheDictionary['DependentData']
            equation.dataCache.allDataCacheDictionary['Weights'] =  []
            equation.Solve()
            equation.CalculateModelErrors(equation.solvedCoefficients, equation.dataCache.allDataCacheDictionary)
            equation.CalculateCoefficientAndFitStatistics()

            fileName = specificBaseName + '_ci' + extraZerosString + str(i) + '_large.png'
            SaveModelScatterConfidence(fileName, equation, Ymax, Ymin)
            
            # small animation, create fewer frames
            if not (i % smallAnimationModulus):
                graphWidth = smallAnimationGraphWidth
                graphHeight = smallAnimationGraphHeight
                fileName = specificBaseName + '_ci' + extraZerosString + str(i) + '_small.png'
                SaveModelScatterConfidence(fileName, equation, Ymax, Ymin)
                
    print
    # convert all PNG files to GIF for gifsicle to use
    commandString = 'mogrify -format gif *png'
    print "Calling " + commandString
    os.popen(commandString)
    
    # make small GIF animation
    commandString = 'gifsicle --loop --colors 256 --delay ' + str(smallAnimationDelayBetweenFrames) + " " + specificBaseName + '_ci*small.gif > ' + specificBaseName + '_small.gif'
    print "Calling " + commandString
    os.popen(commandString)

    # make large GIF animation        
    commandString = 'gifsicle --loop --colors 256 --delay ' + str(largeAnimationDelayBetweenFrames) + " " + specificBaseName + '_ci*large.gif > ' + specificBaseName + '_large.gif'
    print "Calling " + commandString
    os.popen(commandString)

    # remove unused files, saving the ones in this list
    stillImageFileNameList = [specificBaseName + '_ci022_large.png',
                              specificBaseName + '_ci090_large.png',
                              specificBaseName + '_ci270_large.png']
    currentDir = os.listdir('.')
    for filename in currentDir:
        if (-1 != filename.find('_ci')) and (-1 != filename.find('small')):
            os.remove(filename)
        if (-1 != filename.find('_ci')) and (-1 != filename.find('large.gif')):
            os.remove(filename)
        if (-1 != filename.find(specificBaseName)) and (-1 != filename.find('_large.png')) and (filename not in stillImageFileNameList):
            os.remove(filename)

    return [htmlTitle, htmlText, specificBaseName, stillImageFileNameList]

Example 135

Project: multiprocess
Source File: ex_pool.py
View license
def test():
    print 'cpuCount() = %d\n' % cpuCount()
    
    #
    # Create pool
    #
    
    PROCESSES = 4
    print 'Creating pool with %d processes\n' % PROCESSES
    pool = Pool(PROCESSES)    

    #
    # Tests
    #

    TASKS = [(mul, (i, 7)) for i in range(10)] + \
            [(plus, (i, 8)) for i in range(10)]

    results = [pool.apply_async(calculate, t) for t in TASKS]
    imap_it = pool.imap(calculatestar, TASKS)
    imap_unordered_it = pool.imap_unordered(calculatestar, TASKS)

    print 'Ordered results using pool.apply_async():'
    for r in results:
        print '\t', r.get()
    print

    print 'Ordered results using pool.imap():'        
    for x in imap_it:
        print '\t', x
    print

    print 'Unordered results using pool.imap_unordered():'
    for x in imap_unordered_it:
        print '\t', x
    print

    print 'Ordered results using pool.map() --- will block till complete:'
    for x in pool.map(calculatestar, TASKS):
        print '\t', x
    print    

    #
    # Simple benchmarks
    #

    N = 100000
    print 'def pow3(x): return x**3'
    
    t = time.time()
    A = map(pow3, xrange(N))
    print '\tmap(pow3, xrange(%d)):\n\t\t%s seconds' % \
          (N, time.time() - t)
    
    t = time.time()
    B = pool.map(pow3, xrange(N))
    print '\tpool.map(pow3, xrange(%d)):\n\t\t%s seconds' % \
          (N, time.time() - t)

    t = time.time()
    C = list(pool.imap(pow3, xrange(N), chunksize=N//8))
    print '\tlist(pool.imap(pow3, xrange(%d), chunksize=%d)):\n\t\t%s' \
          ' seconds' % (N, N//8, time.time() - t)
    
    assert A == B == C, (len(A), len(B), len(C))
    print
    
    L = [None] * 1000000
    print 'def noop(x): pass'
    print 'L = [None] * 1000000'
    
    t = time.time()
    A = map(noop, L)
    print '\tmap(noop, L):\n\t\t%s seconds' % \
          (time.time() - t)
    
    t = time.time()
    B = pool.map(noop, L)
    print '\tpool.map(noop, L):\n\t\t%s seconds' % \
          (time.time() - t)

    t = time.time()
    C = list(pool.imap(noop, L, chunksize=len(L)//8))
    print '\tlist(pool.imap(noop, L, chunksize=%d)):\n\t\t%s seconds' % \
          (len(L)//8, time.time() - t)

    assert A == B == C, (len(A), len(B), len(C))
    print    

    del A, B, C, L

    #
    # Test error handling
    #

    print 'Testing error handling:'

    try:
        print pool.apply(f, (5,))
    except ZeroDivisionError:
        print '\tGot ZeroDivisionError as expected from pool.apply()'
    else:
        raise AssertionError, 'expected ZeroDivisionError'

    try:
        print pool.map(f, range(10))
    except ZeroDivisionError:
        print '\tGot ZeroDivisionError as expected from pool.map()'
    else:
        raise AssertionError, 'expected ZeroDivisionError'
            
    try:
        print list(pool.imap(f, range(10)))
    except ZeroDivisionError:
        print '\tGot ZeroDivisionError as expected from list(pool.imap())'
    else:
        raise AssertionError, 'expected ZeroDivisionError'

    it = pool.imap(f, range(10))
    for i in range(10):
        try:
            x = it.next()
        except ZeroDivisionError:
            if i == 5:
                pass
        except StopIteration:
            break
        else:
            if i == 5:
                raise AssertionError, 'expected ZeroDivisionError'
            
    assert i == 9
    print '\tGot ZeroDivisionError as expected from IMapIterator.next()'
    print
    
    #
    # Testing timeouts
    #
    
    print 'Testing ApplyResult.get() with timeout:',
    res = pool.apply_async(calculate, TASKS[0])
    while 1:
        sys.stdout.flush()
        try:
            sys.stdout.write('\n\t%s' % res.get(0.02))
            break
        except TimeoutError:
            sys.stdout.write('.')
    print
    print

    print 'Testing IMapIterator.next() with timeout:',
    it = pool.imap(calculatestar, TASKS)
    while 1:
        sys.stdout.flush()
        try:
            sys.stdout.write('\n\t%s' % it.next(0.02))
        except StopIteration:
            break
        except TimeoutError:
            sys.stdout.write('.')
    print
    print
            
    #
    # Testing callback
    #

    print 'Testing callback:'
    
    A = []
    B = [56, 0, 1, 8, 27, 64, 125, 216, 343, 512, 729]
        
    r = pool.apply_async(mul, (7, 8), callback=A.append)
    r.wait()

    r = pool.map_async(pow3, range(10), callback=A.extend)
    r.wait()

    if A == B:
        print '\tcallbacks succeeded\n'
    else:
        print '\t*** callbacks failed\n\t\t%s != %s\n' % (A, B)
    
    #
    # Check there are no outstanding tasks
    #
    
    assert not pool._cache, 'cache = %r' % pool._cache

    #
    # Check close() methods
    #

    print 'Testing close():'

    for worker in pool._pool:
        assert worker.is_alive()

    result = pool.apply_async(time.sleep, [0.5])
    pool.close()
    pool.join()

    assert result.get() is None

    for worker in pool._pool:
        assert not worker.is_alive()

    print '\tclose() succeeded\n'

    #
    # Check terminate() method
    #

    print 'Testing terminate():'

    pool = Pool(2)
    ignore = pool.apply(pow3, [2])
    results = [pool.apply_async(time.sleep, [10]) for i in range(10)]
    pool.terminate()
    pool.join()

    for worker in pool._pool:
        assert not worker.is_alive()

    print '\tterminate() succeeded\n'

    #
    # Check garbage collection
    #

    print 'Testing garbage collection:'

    pool = Pool(2)
    processes = pool._pool
    
    ignore = pool.apply(pow3, [2])
    results = [pool.apply_async(time.sleep, [10]) for i in range(10)]

    del results, pool

    time.sleep(0.2)
    
    for worker in processes:
        assert not worker.is_alive()

    print '\tgarbage collection succeeded\n'

Example 136

Project: pycog
Source File: sgd.py
View license
    def train(self, gradient_data, validation_data, savefile):
        """
        Train the RNN.

        Paramters
        ---------

        gradient_data : pycog.Dataset
                        Gradient dataset.

        validation_data : pycog.Dataset
                          Validation dataset.

        savefile : str
                   File to save network information in.

        """
        checkfreq = self.p['checkfreq']
        if checkfreq is None:
            checkfreq = int(1e4)//gradient_data.minibatch_size

        patience = self.p['patience']
        if patience is None:
            patience = 100*checkfreq

        alpha        = self.p['dt']/self.p['tau']
        lambda_Omega = self.p['lambda_Omega']
        lr           = self.p['learning_rate']
        maxnorm      = self.p['max_gradient_norm']
        bound        = self.p['bound']
        save_exclude = ['callback', 'performance', 'terminate']

        #---------------------------------------------------------------------------------
        # Continue previous run if we can
        #---------------------------------------------------------------------------------

        if os.path.isfile(savefile):
            with open(savefile) as f:
                save = pickle.load(f)
            best          = save['best']
            init_p        = save['current']
            first_iter    = save['iter']
            costs_history = save['costs_history']
            Omega_history = save['Omega_history']

            # Restore RNGs for datasets
            gradient_data.rng   = save['rng_gradient']
            validation_data.rng = save['rng_validation']

            # Restore parameter values
            for i, j in zip(self.trainables, init_p):
                i.set_value(j)

            print(("[ {}.SGD.train ] Recovered saved model,"
                   " continuing from iteration {}.").format(THIS, first_iter))
        else:
            best = {
                'iter':        1,
                'cost':        np.inf,
                'other_costs': [],
                'params':      SGD.get_values(self.save_values)
                }
            first_iter    = best['iter']
            costs_history = []
            Omega_history = []

            # Save initial conditions
            save = {
                'params':         {k: v for k, v in self.p.items()
                                   if k not in save_exclude},
                'varlist':        self.trainable_names,
                'iter':           1,
                'current':        SGD.get_values(self.trainables),
                'best':           best,
                'costs_history':  costs_history,
                'Omega_history':  Omega_history,
                'rng_gradient':   gradient_data.rng,
                'rng_validation': validation_data.rng
                }
            base, ext = os.path.splitext(savefile)
            dump(base + '_init' + ext, save)

        #---------------------------------------------------------------------------------
        # Updates
        #---------------------------------------------------------------------------------

        performance = self.p['performance']
        terminate   = self.p['terminate']
        tr_Omega    = None
        tr_gnorm    = None
        try:
            tstart = datetime.datetime.now()
            for iter in xrange(first_iter, 1+self.p['max_iter']):
                if iter % checkfreq == 1:
                    #---------------------------------------------------------------------
                    # Timestamp
                    #---------------------------------------------------------------------

                    tnow      = datetime.datetime.now()
                    totalsecs = (tnow - tstart).total_seconds()

                    hrs  = int(totalsecs//3600)
                    mins = int(totalsecs%3600)//60
                    secs = int(totalsecs%60)

                    timestamp = tnow.strftime('%b %d %Y %I:%M:%S %p').replace(' 0', ' ')
                    print('{} updates - {} ({} hrs {} mins {} secs elapsed)'
                          .format(iter-1, timestamp, hrs, mins, secs))

                    #---------------------------------------------------------------------
                    # Validate
                    #---------------------------------------------------------------------

                    # Validation cost
                    costs = self.f_cost(*validation_data(best['other_costs']))
                    z     = costs[-1] # network outputs
                    costs = [float(i) for i in costs[:-1]]
                    s0    = "| validation loss / RMSE"
                    s1    = ": {:.6f} / {:.6f}".format(costs[0], costs[1])

                    # Dashes
                    nfill = 70

                    # Compute task-specific performance
                    if performance is not None:
                        costs.append(performance(validation_data.get_trials(),
                                                 SGD.get_value(z)))
                        s0    += " / performance"
                        s1    += " / {:.2f}".format(costs[-1])
                    s = s0 + s1

                    # Callback
                    if self.p['callback'] is not None:
                        callback_results = self.p['callback'](
                            validation_data.get_trials(), SGD.get_value(z)
                            )
                    else:
                        callback_results = None

                    # Keep track of costs
                    costs_history.append((gradient_data.ntrials, costs))

                    # Record the value of the regularization term in the last iteration
                    if tr_Omega is not None:
                        Omega_history.append(
                            (gradient_data.ntrials, lambda_Omega*tr_Omega)
                            )

                    # New best
                    if costs[0] < best['cost']:
                        s += ' ' + '-'*(nfill - len(s))
                        s += " NEW BEST (prev. best: {:.6f})".format(best['cost'])
                        best = {
                            'iter':        iter,
                            'cost':        costs[0],
                            'other_costs': costs[1:],
                            'params':      SGD.get_values(self.save_values)
                            }
                    print(s)

                    # Spectral radius
                    rho = RNN.spectral_radius(self.Wrec_.eval())

                    # Format
                    Omega = ('n/a' if tr_Omega is None
                             else '{:.8f}'.format(float(tr_Omega)))
                    gnorm = ('n/a' if tr_gnorm is None
                             else '{:.8f}'.format(float(tr_gnorm)))

                    # Info
                    print("| Omega      (last iter) = {}".format(Omega))
                    print("| grad. norm (last iter) = {}".format(gnorm))
                    print("| rho                    = {:.8f}".format(rho))
                    sys.stdout.flush()

                    #---------------------------------------------------------------------
                    # Save progress
                    #---------------------------------------------------------------------

                    save = {
                        'params':         {k: v for k, v in self.p.items()
                                           if k not in save_exclude},
                        'varlist':        self.trainable_names,
                        'iter':           iter,
                        'current':        SGD.get_values(self.trainables),
                        'best':           best,
                        'costs_history':  costs_history,
                        'Omega_history':  Omega_history,
                        'rng_gradient':   gradient_data.rng,
                        'rng_validation': validation_data.rng
                        }
                    dump(savefile, save)

                    if costs[1] <= self.p['min_error']:
                        print("Reached minimum error of {:.6f}"
                              .format(self.p['min_error']))
                        break

                    # This termination criterion assumes that performance is not None
                    if terminate(np.array([c[-1] for _, c in costs_history])):
                        print("Termination criterion satisfied -- we\'ll call it a day.")
                        break

                if iter - best['iter'] > patience:
                    print("We've run out of patience -- time to give up.")
                    break

                #-------------------------------------------------------------------------
                # Training step
                #-------------------------------------------------------------------------

                tr_cost, tr_gnorm, tr_Omega, tr_nelems, tr_x = self.train_step(
                    *(gradient_data(best['other_costs'], callback_results)
                      + [alpha, lambda_Omega, lr, maxnorm, bound])
                     )

                #-------------------------------------------------------------------------
        except KeyboardInterrupt:
            print("[ {}.SGD.train ] Training interrupted by user during iteration {}."
                  .format(THIS, iter))

Example 137

Project: sql2graph
Source File: export.py
View license
    def export_tabledump(self, entity, fp,
            nodes_writer, rels_writer, index_writers):

        stats = {'nodes': 0, 'rels': 0, 'indexed': 0}
        if not entity:
            print "know nothing about %s" % entity.name
            return

        PRINT_FREQUENCY = 25000

        # should we write something to one or more indexes?
        if index_writers:
            indexes = entity.get_indexed_fields()
        else:
            indexes = None

        node_id = 0

        # read CSV file line by line
        #print self.dialect
        csvreader = csv.DictReader(fp, dialect=self.dialect)
        for cnt, record in enumerate(csvreader, start=1):

            node = None

            # create a new node
            primary_key_field = entity.get_primary_key_field()
            if primary_key_field:

                node = graph.Node(record, entity)
                node_id = self.node_list.add_node(node)
                if not node_id:
                    # FIXME: find something better
                    raise LookupError

                # add it to the write queue
                nodes_writer.append(node.get_dict(self.nodes_csv_fields))

                stats['nodes'] += 1

                if indexes:
                    for index_name, indexed_fields in indexes.iteritems():
                        index_writers.get(index_name).append(
                            node.get_dict(
                                ['node_id'] + [field.name for field in indexed_fields]))
                        stats['indexed'] += 1

            # add relations if needed
            new_rels = [rel.get_dict(self.rels_csv_fields)
                for rel in self.iter_relations(entity, record)]
            rels_writer.extend(new_rels)
            stats['rels'] += len(new_rels)

            # hint to gc; there's surely something prettier
            if node:
                del node
            del record

            if self.DEBUG:
                if not (cnt % PRINT_FREQUENCY):
                    print "\r line %8d - nodes(%8d), rels(%8d), idx(%8d) -- last node ID %d" % (
                        cnt, stats['nodes'], stats['rels'], stats['indexed'], node_id),
                    sys.stdout.flush()

        if self.DEBUG:
            print
            print " --> Finished with %8d of entity %s" % (cnt, entity.name)
            print "nodes(%8d), rels(%8d), idx(%8d) -- last node ID %d" % (
                        stats['nodes'], stats['rels'], stats['indexed'],  node_id)

        # write everything that's pending
        writers = [nodes_writer, rels_writer] + list(index_writers.itervalues())
        for w in writers:
            if w:
                w.flush()

Example 138

Project: Rusthon
Source File: run.py
View license
def run_test_on(filename):
    """run one test and returns the number of errors"""
    if not show_details:
        f = open(filename)
        comment = f.readline().strip(" \n\"'")
        f.close()
        print(table_header % (filename[2:-3], comment), end='')
    sum_errors = {}

    if filename.endswith('.html'):
        run_html_test( filename, sum_errors )
        return sum_errors

    def display(function):
        global _test_description
        _test_description = function.__doc__
        if show_details:
            print('\n<%s>\n' % function.__doc__)

        errors = function(filename)
        if errors:
            if not show_details:
                print(table_cell % ''.join('%s%d' % (k[0], v)
                                            for k, v in errors.items()),
                      end='')
        else:
            if not show_details:
                print(table_cell % 'OK', end='')
        sys.stdout.flush()

        for k, v in errors.items():
            sum_errors[k] = sum_errors.get(k, 0) + v

        if show_details:
            print('-'*77)

    if 'requirejs' not in filename and not filename.startswith('./go/'):
        display(run_python_test_on)
        display(run_python3_test_on)
        if pypy_runnable:
            display(run_pypy_test_on)
        if old_pypy_runnable:
            display(run_old_pypy_test_on)

    global js
    do_js_test = not filename.startswith( ('./go/', './asm/', './rust/', './c++') )
    if do_js_test:
        js = translate_js(
            filename,
            javascript=True,
            multioutput=filename.startswith('./threads/' or filename.startswith('./bench/webworker'))
        )
        if rhino_runnable:
            display(run_pythonjs_test_on)
        if node_runnable:
            display(run_pythonjs_test_on_node)

        if nodewebkit_runnable:
            display(run_pythonjs_test_on_nodewebkit)

    ## TODO more optimized js settings pythonjs-minimal.js ##
    #if '--no-javascript-mode' not in sys.argv and do_js_test:
    #    js = translate_js(filename, javascript=True, multioutput=filename.startswith('./threads/' or filename.startswith('./bench/webworker')))
    #    if rhino_runnable:
    #        display(run_pythonjsjs_test_on)
    #    if node_runnable:
    #        display(run_pythonjsjs_test_on_node)
    #    if nodewebkit_runnable:
    #        display(run_pythonjsjs_test_on_nodewebkit)


    if 'requirejs' not in filename:

        if dart_runnable:
            js = translate_js(filename, javascript=False, dart=True)
            display(run_pythonjs_dart_test_on_dart)

        if dart2js_runnable and node_runnable:
            js = translate_js(filename, javascript=False, dart=True)
            display(run_pythonjs_dart_test_on_node)

        if coffee_runnable and node_runnable:
            js = translate_js(filename, javascript=False, dart=False, coffee=True)
            display(run_pythonjs_coffee_test_on_node)

        if luajs_runnable and node_runnable:
            js = translate_js(filename, luajs=True)
            display(run_pythonjs_luajs_test_on_node)

        if lua_runnable:
            js = translate_js(filename, lua=True)
            display(run_pythonjs_lua_test_on_lua)

        if luajit_runnable:
            js = translate_js(filename, lua=True)
            display(run_pythonjs_lua_test_on_luajit)

        if go_runnable:
            js = translate_js(filename, go=True)
            display(run_pythonjs_go_test)

        if gopherjs_runnable:
            js = translate_js(filename, gopherjs=True)
            display(run_pythonjs_gopherjs_test)

        if rust_runnable:
            js = translate_js(filename, rust=True)
            display(run_pythonjs_rust_test)

        if cpp_runnable:
            js = translate_js(filename, cpp=True)
            display(run_pythonjs_cpp_test)


    print()
    return sum_errors

Example 139

Project: theanolm
Source File: basictrainer.py
View license
    def __init__(self, training_options, optimization_options,
                 network, vocabulary, scorer,
                 training_files, sampling, validation_iter, state,
                 profile=False):
        """Creates the optimizer and initializes the training process.

        :type training_options: dict
        :param training_options: a dictionary of training options

        :type optimization_options: dict
        :param optimization_options: a dictionary of optimization options

        :type network: Network
        :param network: a neural network to be trained

        :type vocabulary: Vocabulary
        :param vocabulary: vocabulary that provides mapping between words and
                           word IDs

        :type scorer: TextScorer
        :param scorer: a text scorer for computing validation set perplexity

        :type training_files: list of file objects
        :param training_files: list of files to be used as training data

        :type sampling: list of floats
        :param sampling: specifies a fraction for each training file, how much
                         to sample on each epoch

        :type validation_iter: theanolm.BatchIterator
        :param validation_iter: an iterator for computing validation set
                                perplexity

        :type state: h5py.File
        :param state: HDF5 file where initial training state will be possibly
                      read from, and candidate states will be saved to

        :type profile: bool
        :param profile: if set to True, creates Theano profile objects
        """

        self.network = network
        self.vocabulary = vocabulary
        self.scorer = scorer
        self.validation_iter = validation_iter

        self.optimizer = create_optimizer(optimization_options,
                                          self.network,
                                          profile)

        self.training_iter = ShufflingBatchIterator(
            training_files,
            sampling,
            vocabulary,
            batch_size=training_options['batch_size'],
            max_sequence_length=training_options['sequence_length'])

        print("Computing the number of training updates per epoch.")
        sys.stdout.flush()
        self.updates_per_epoch = len(self.training_iter)
        if self.updates_per_epoch < 1:
            raise ValueError("Training data does not contain any sentences.")

        self.stopper = create_stopper(training_options, self)
        self.options = training_options

        # current candidate for the minimum validation cost state
        self._candidate_state = state
        if 'trainer' in self._candidate_state:
            print("Restoring initial network state from {}.".format(
                self._candidate_state.filename))
            sys.stdout.flush()
            self._reset_state()
        else:
            # index to the cost history that corresponds to the current candidate
            # state
            self._candidate_index = None
            # current training epoch
            self.epoch_number = 1
            # number of mini-batch updates performed in this epoch
            self.update_number = 0
            # validation set cost history
            self._cost_history = numpy.asarray([], dtype=theano.config.floatX)

        # number of mini-batch updates between log messages
        self.log_update_interval = 0
        # total number of mini-batch updates performed (after restart)
        self.total_updates = 0

Example 140

Project: multiprocess
Source File: ex_pool.py
View license
def test():
    print('cpuCount() = %d\n' % cpuCount())
    
    #
    # Create pool
    #
    
    PROCESSES = 4
    print('Creating pool with %d processes\n' % PROCESSES)
    pool = Pool(PROCESSES)    

    #
    # Tests
    #

    TASKS = [(mul, (i, 7)) for i in range(10)] + \
            [(plus, (i, 8)) for i in range(10)]

    results = [pool.apply_async(calculate, t) for t in TASKS]
    imap_it = pool.imap(calculatestar, TASKS)
    imap_unordered_it = pool.imap_unordered(calculatestar, TASKS)

    print('Ordered results using pool.apply_async():')
    for r in results:
        print('\t', r.get())
    print()

    print('Ordered results using pool.imap():')
    for x in imap_it:
        print('\t', x)
    print()

    print('Unordered results using pool.imap_unordered():')
    for x in imap_unordered_it:
        print('\t', x)
    print()

    print('Ordered results using pool.map() --- will block till complete:')
    for x in pool.map(calculatestar, TASKS):
        print('\t', x)
    print()

    #
    # Simple benchmarks
    #

    N = 100000
    print('def pow3(x): return x**3')
    
    t = time.time()
    A = list(map(pow3, xrange(N)))
    print('\tmap(pow3, xrange(%d)):\n\t\t%s seconds' % \
          (N, time.time() - t))
    
    t = time.time()
    B = pool.map(pow3, xrange(N))
    print('\tpool.map(pow3, xrange(%d)):\n\t\t%s seconds' % \
          (N, time.time() - t))

    t = time.time()
    C = list(pool.imap(pow3, xrange(N), chunksize=N//8))
    print('\tlist(pool.imap(pow3, xrange(%d), chunksize=%d)):\n\t\t%s' \
          ' seconds' % (N, N//8, time.time() - t))
    
    assert A == B == C, (len(A), len(B), len(C))
    print()
    
    L = [None] * 1000000
    print('def noop(x): pass')
    print('L = [None] * 1000000')
    
    t = time.time()
    A = list(map(noop, L))
    print('\tmap(noop, L):\n\t\t%s seconds' % \
          (time.time() - t))
    
    t = time.time()
    B = pool.map(noop, L)
    print('\tpool.map(noop, L):\n\t\t%s seconds' % \
          (time.time() - t))

    t = time.time()
    C = list(pool.imap(noop, L, chunksize=len(L)//8))
    print('\tlist(pool.imap(noop, L, chunksize=%d)):\n\t\t%s seconds' % \
          (len(L)//8, time.time() - t))

    assert A == B == C, (len(A), len(B), len(C))
    print()

    del A, B, C, L

    #
    # Test error handling
    #

    print('Testing error handling:')

    try:
        print(pool.apply(f, (5,)))
    except ZeroDivisionError:
        print('\tGot ZeroDivisionError as expected from pool.apply()')
    else:
        raise AssertionError('expected ZeroDivisionError')

    try:
        print(pool.map(f, range(10)))
    except ZeroDivisionError:
        print('\tGot ZeroDivisionError as expected from pool.map()')
    else:
        raise AssertionError('expected ZeroDivisionError')
            
    try:
        print(list(pool.imap(f, range(10))))
    except ZeroDivisionError:
        print('\tGot ZeroDivisionError as expected from list(pool.imap())')
    else:
        raise AssertionError('expected ZeroDivisionError')

    it = pool.imap(f, range(10))
    for i in range(10):
        try:
            x = it.next()
        except ZeroDivisionError:
            if i == 5:
                pass
        except StopIteration:
            break
        else:
            if i == 5:
                raise AssertionError('expected ZeroDivisionError')
            
    assert i == 9
    print('\tGot ZeroDivisionError as expected from IMapIterator.next()')
    print()
    
    #
    # Testing timeouts
    #
    
    print('Testing ApplyResult.get() with timeout:', end='')
    res = pool.apply_async(calculate, TASKS[0])
    while 1:
        sys.stdout.flush()
        try:
            sys.stdout.write('\n\t%s' % res.get(0.02))
            break
        except TimeoutError:
            sys.stdout.write('.')
    print()
    print()

    print('Testing IMapIterator.next() with timeout:', end='')
    it = pool.imap(calculatestar, TASKS)
    while 1:
        sys.stdout.flush()
        try:
            sys.stdout.write('\n\t%s' % it.next(0.02))
        except StopIteration:
            break
        except TimeoutError:
            sys.stdout.write('.')
    print()
    print()
            
    #
    # Testing callback
    #

    print('Testing callback:')
    
    A = []
    B = [56, 0, 1, 8, 27, 64, 125, 216, 343, 512, 729]
        
    r = pool.apply_async(mul, (7, 8), callback=A.append)
    r.wait()

    r = pool.map_async(pow3, range(10), callback=A.extend)
    r.wait()

    if A == B:
        print('\tcallbacks succeeded\n')
    else:
        print('\t*** callbacks failed\n\t\t%s != %s\n' % (A, B))
    
    #
    # Check there are no outstanding tasks
    #
    
    assert not pool._cache, 'cache = %r' % pool._cache

    #
    # Check close() methods
    #

    print('Testing close():')

    for worker in pool._pool:
        assert worker.is_alive()

    result = pool.apply_async(time.sleep, [0.5])
    pool.close()
    pool.join()

    assert result.get() is None

    for worker in pool._pool:
        assert not worker.is_alive()

    print('\tclose() succeeded\n')

    #
    # Check terminate() method
    #

    print('Testing terminate():')

    pool = Pool(2)
    ignore = pool.apply(pow3, [2])
    results = [pool.apply_async(time.sleep, [10]) for i in range(10)]
    pool.terminate()
    pool.join()

    for worker in pool._pool:
        assert not worker.is_alive()

    print('\tterminate() succeeded\n')

    #
    # Check garbage collection
    #

    print('Testing garbage collection:')

    pool = Pool(2)
    processes = pool._pool
    
    ignore = pool.apply(pow3, [2])
    results = [pool.apply_async(time.sleep, [10]) for i in range(10)]

    del results, pool

    time.sleep(0.2)
    
    for worker in processes:
        assert not worker.is_alive()

    print('\tgarbage collection succeeded\n')

Example 141

View license
def main():
    with open(filename) as bookmarks_file:
        data = bookmarks_file.read()
    
    geolocator = Nominatim()

    kml = simplekml.Kml()

    lst = list()
    
    # Hacky and doesn't work for all of the stars:
    lat_re = re.compile('markers:[^\]]*latlng[^}]*lat:([^,]*)')
    lon_re = re.compile('markers:[^\]]*latlng[^}]*lng:([^}]*)')
    coords_in_url = re.compile('\?q=(-?\d{,3}\.\d*),\s*(-?\d{,3}\.\d*)')
    
    doc = document_fromstring(data)
    for element, attribute, url, pos in doc.body.iterlinks():
        if 'maps.google' in url:
            description = element.text or ''
            print description.encode('UTF8')
            
            if coords_in_url.search(url):
                # Coordinates are in URL itself
                latitude = coords_in_url.search(url).groups()[0]
                longitude = coords_in_url.search(url).groups()[1]
            else:
                # Load map and find coordinates in source of page
                try:
                    sock = urlopen(url.replace(' ','+').encode('UTF8'))
                except Exception, e:
                    print 'Connection problem:'
                    print repr(e)
                    print 'Waiting 2 minutes and trying again'
                    time.sleep(120)
                    sock = urlopen(url.replace(' ','+').encode('UTF8'))
                content = sock.read()
                sock.close()
                time.sleep(3) # Don't annoy server
                try:
                    latitude = lat_re.findall(content)[0]
                    longitude = lon_re.findall(content)[0]
                except IndexError:
                    try:
                        lines = content.split('\n')  # --> ['Line 1', 'Line 2', 'Line 3']
                        for line in lines:
                            if re.search('cacheResponse\(', line):
                                splitline = line.split('(')[1].split(')')[0] + '"]'
                                # in the future we can extract the coordinates from here
                                null = None
                                values = eval(splitline)
                                print values[8][0][1]
                                longitude = str(values[0][0][1])
                                latitude = str(values[0][0][2])
                                continue
                    except IndexError:
                        print '[Coordinates not found]'
                        continue
                    print
            
            print latitude, longitude
            try:
    			location = geolocator.reverse(latitude+", "+longitude)
    			print(location.address)
            except ValueError:
                print '[Invalid coordinates]'
            print
            kml.newpoint(name=description, coords=[(float(longitude), float(latitude))])
            lst.append({'latitude': latitude,
                       'longitude': longitude,
                       'name': description,
                       'url': url.encode(encoding='utf-8', errors='replace'),
                       'address': location.address.encode(encoding='utf-8', errors='replace') if location else 'error'})

            # this is here because there's a tendancy for this script to fail part way through...
            # so at least you can get a partial result
            kml.save("GoogleBookmarks.kml")
            with open('GoogleBookmarks.json', mode='w') as listdump:
                listdump.write(json.dumps(lst))
        sys.stdout.flush()

    kml.save("GoogleBookmarks.kml")
    with open('GoogleBookmarks.json', mode='w') as listdump:
        listdump.write(json.dumps(lst))

Example 142

Project: pyeq3
Source File: generateOutput.py
View license
def RandomData_A():
    specificBaseName = inspect.stack()[0][3]
    htmlTitle = 'Fitting Random Data'
    htmlText = '''
This illustrates the effect of fitting completely<br>
random data that has no relationship of any kind.'''
    thumbnailAnimationName = specificBaseName + '_small.gif'
    fullsizeAnimationName = specificBaseName + '_large.gif'
    
    print(specificBaseName)

    Ymax = 1.0
    Ymin = 0.0
    
    equation = pyeq3.Models_2D.Polynomial.Linear('SSQABS')
    dimensionality = equation.GetDimensionality()
    equation.dataCache.DependentDataContainsZeroFlag = True # do not need relative error calculations, this flag turns them off
        
    numpy.random.seed(3) # yield repeatable results
    randomArrayX = numpy.random.random_sample(195)
    randomArrayY = numpy.random.random_sample(195)
    equation.dataCache.allDataCacheDictionary['Weights'] = []
    
    # ensure a X data range for the graph, as on
    # all other plots the X data range is constant
    randomArrayX[0] = 0.001
    randomArrayX[1] = 0.999
    
    for i in range(0, 360):
        extraZerosString = ''
        if i < 100:
            extraZerosString = '0'
        if i < 10:
            extraZerosString = '00'
        
        if not i % largeAnimationModulus:
            print(i, end=' ');sys.stdout.flush()
            index = i
            if i > 180:
                index = 180 - (i%180)
            equation = pyeq3.Models_2D.Polynomial.Linear('SSQABS')
            xdata = randomArrayX[:15 + index/2]
            ydata = randomArrayY[:15 + index/2]
            equation.dataCache.allDataCacheDictionary['IndependentData'] = numpy.array([xdata, numpy.ones(len(xdata))])
            IndependentDataArray = equation.dataCache.allDataCacheDictionary['IndependentData']
            equation.dataCache.allDataCacheDictionary['DependentData'] =  ydata
            DependentDataArray = equation.dataCache.allDataCacheDictionary['DependentData']
            equation.dataCache.allDataCacheDictionary['Weights'] =  []
            equation.Solve()
            equation.CalculateModelErrors(equation.solvedCoefficients, equation.dataCache.allDataCacheDictionary)
            equation.CalculateCoefficientAndFitStatistics()
            
            fileName = specificBaseName + '_ci' + extraZerosString + str(i) + '_large.png'
            SaveModelScatterConfidence(fileName, equation, Ymax, Ymin)
            
            # small animation, create fewer frames
            if not (i % smallAnimationModulus):
                graphWidth = smallAnimationGraphWidth
                graphHeight = smallAnimationGraphHeight
                fileName = specificBaseName + '_ci' + extraZerosString + str(i) + '_small.png'
                SaveModelScatterConfidence(fileName, equation, Ymax, Ymin)
                
    print()
    # convert all PNG files to GIF for gifsicle to use
    commandString = 'mogrify -format gif *png'
    print("Calling " + commandString)
    p = os.popen(commandString)
    p.close()
    
    # make small GIF animation
    commandString = 'gifsicle --loop --colors 256 --delay ' + str(smallAnimationDelayBetweenFrames) + " " + specificBaseName + '_ci*small.gif > ' + specificBaseName + '_small.gif'
    print("Calling " + commandString)
    p = os.popen(commandString)
    p.close()

    # make large GIF animation        
    commandString = 'gifsicle --loop --colors 256 --delay ' + str(largeAnimationDelayBetweenFrames) + " " + specificBaseName + '_ci*large.gif > ' + specificBaseName + '_large.gif'
    print("Calling " + commandString)
    p = os.popen(commandString)
    p.close()

    # remove unused files, saving the ones in this list
    stillImageFileNameList = [specificBaseName + '_ci000_large.png',
                              specificBaseName + '_ci180_large.png']
    currentDir = os.listdir('.')
    for filename in currentDir:
        if (-1 != filename.find('_ci')) and (-1 != filename.find('small')):
            os.remove(filename)
        if (-1 != filename.find('_ci')) and (-1 != filename.find('large.gif')):
            os.remove(filename)
        if (-1 != filename.find(specificBaseName)) and (-1 != filename.find('_large.png')) and (filename not in stillImageFileNameList):
            os.remove(filename)

    return [htmlTitle, htmlText, specificBaseName, stillImageFileNameList]

Example 143

Project: sshuttle
Source File: client.py
View license
def _main(tcp_listener, udp_listener, fw, ssh_cmd, remotename,
          python, latency_control,
          dns_listener, seed_hosts, auto_nets, daemon):

    debug1('Starting client with Python version %s\n'
           % platform.python_version())

    method = fw.method

    handlers = []
    if helpers.verbose >= 1:
        helpers.logprefix = 'c : '
    else:
        helpers.logprefix = 'client: '
    debug1('connecting to server...\n')

    try:
        (serverproc, serversock) = ssh.connect(
            ssh_cmd, remotename, python,
            stderr=ssyslog._p and ssyslog._p.stdin,
            options=dict(latency_control=latency_control))
    except socket.error as e:
        if e.args[0] == errno.EPIPE:
            raise Fatal("failed to establish ssh session (1)")
        else:
            raise
    mux = Mux(serversock, serversock)
    handlers.append(mux)

    expected = b'SSHUTTLE0001'

    try:
        v = 'x'
        while v and v != b'\0':
            v = serversock.recv(1)
        v = 'x'
        while v and v != b'\0':
            v = serversock.recv(1)
        initstring = serversock.recv(len(expected))
    except socket.error as e:
        if e.args[0] == errno.ECONNRESET:
            raise Fatal("failed to establish ssh session (2)")
        else:
            raise

    rv = serverproc.poll()
    if rv:
        raise Fatal('server died with error code %d' % rv)

    if initstring != expected:
        raise Fatal('expected server init string %r; got %r'
                    % (expected, initstring))
    log('Connected.\n')
    sys.stdout.flush()
    if daemon:
        daemonize()
        log('daemonizing (%s).\n' % _pidname)

    def onroutes(routestr):
        if auto_nets:
            for line in routestr.strip().split(b'\n'):
                (family, ip, width) = line.split(b',', 2)
                family = int(family)
                width = int(width)
                ip = ip.decode("ASCII")
                if family == socket.AF_INET6 and tcp_listener.v6 is None:
                    debug2("Ignored auto net %d/%s/%d\n" % (family, ip, width))
                if family == socket.AF_INET and tcp_listener.v4 is None:
                    debug2("Ignored auto net %d/%s/%d\n" % (family, ip, width))
                else:
                    debug2("Adding auto net %d/%s/%d\n" % (family, ip, width))
                    fw.auto_nets.append((family, ip, width))

        # we definitely want to do this *after* starting ssh, or we might end
        # up intercepting the ssh connection!
        #
        # Moreover, now that we have the --auto-nets option, we have to wait
        # for the server to send us that message anyway.  Even if we haven't
        # set --auto-nets, we might as well wait for the message first, then
        # ignore its contents.
        mux.got_routes = None
        fw.start()
    mux.got_routes = onroutes

    def onhostlist(hostlist):
        debug2('got host list: %r\n' % hostlist)
        for line in hostlist.strip().split():
            if line:
                name, ip = line.split(b',', 1)
                fw.sethostip(name, ip)
    mux.got_host_list = onhostlist

    tcp_listener.add_handler(handlers, onaccept_tcp, method, mux)

    if udp_listener:
        udp_listener.add_handler(handlers, onaccept_udp, method, mux)

    if dns_listener:
        dns_listener.add_handler(handlers, ondns, method, mux)

    if seed_hosts is not None:
        debug1('seed_hosts: %r\n' % seed_hosts)
        mux.send(0, ssnet.CMD_HOST_REQ, str.encode('\n'.join(seed_hosts)))

    while 1:
        rv = serverproc.poll()
        if rv:
            raise Fatal('server died with error code %d' % rv)

        ssnet.runonce(handlers, mux)
        if latency_control:
            mux.check_fullness()

Example 144

Project: multiprocess
Source File: ex_pool.py
View license
def test():
    print('cpuCount() = %d\n' % cpuCount())
    
    #
    # Create pool
    #
    
    PROCESSES = 4
    print('Creating pool with %d processes\n' % PROCESSES)
    pool = Pool(PROCESSES)    

    #
    # Tests
    #

    TASKS = [(mul, (i, 7)) for i in range(10)] + \
            [(plus, (i, 8)) for i in range(10)]

    results = [pool.apply_async(calculate, t) for t in TASKS]
    imap_it = pool.imap(calculatestar, TASKS)
    imap_unordered_it = pool.imap_unordered(calculatestar, TASKS)

    print('Ordered results using pool.apply_async():')
    for r in results:
        print('\t', r.get())
    print()

    print('Ordered results using pool.imap():')
    for x in imap_it:
        print('\t', x)
    print()

    print('Unordered results using pool.imap_unordered():')
    for x in imap_unordered_it:
        print('\t', x)
    print()

    print('Ordered results using pool.map() --- will block till complete:')
    for x in pool.map(calculatestar, TASKS):
        print('\t', x)
    print()

    #
    # Simple benchmarks
    #

    N = 100000
    print('def pow3(x): return x**3')
    
    t = time.time()
    A = list(map(pow3, xrange(N)))
    print('\tmap(pow3, xrange(%d)):\n\t\t%s seconds' % \
          (N, time.time() - t))
    
    t = time.time()
    B = pool.map(pow3, xrange(N))
    print('\tpool.map(pow3, xrange(%d)):\n\t\t%s seconds' % \
          (N, time.time() - t))

    t = time.time()
    C = list(pool.imap(pow3, xrange(N), chunksize=N//8))
    print('\tlist(pool.imap(pow3, xrange(%d), chunksize=%d)):\n\t\t%s' \
          ' seconds' % (N, N//8, time.time() - t))
    
    assert A == B == C, (len(A), len(B), len(C))
    print()
    
    L = [None] * 1000000
    print('def noop(x): pass')
    print('L = [None] * 1000000')
    
    t = time.time()
    A = list(map(noop, L))
    print('\tmap(noop, L):\n\t\t%s seconds' % \
          (time.time() - t))
    
    t = time.time()
    B = pool.map(noop, L)
    print('\tpool.map(noop, L):\n\t\t%s seconds' % \
          (time.time() - t))

    t = time.time()
    C = list(pool.imap(noop, L, chunksize=len(L)//8))
    print('\tlist(pool.imap(noop, L, chunksize=%d)):\n\t\t%s seconds' % \
          (len(L)//8, time.time() - t))

    assert A == B == C, (len(A), len(B), len(C))
    print()

    del A, B, C, L

    #
    # Test error handling
    #

    print('Testing error handling:')

    try:
        print(pool.apply(f, (5,)))
    except ZeroDivisionError:
        print('\tGot ZeroDivisionError as expected from pool.apply()')
    else:
        raise AssertionError('expected ZeroDivisionError')

    try:
        print(pool.map(f, range(10)))
    except ZeroDivisionError:
        print('\tGot ZeroDivisionError as expected from pool.map()')
    else:
        raise AssertionError('expected ZeroDivisionError')
            
    try:
        print(list(pool.imap(f, range(10))))
    except ZeroDivisionError:
        print('\tGot ZeroDivisionError as expected from list(pool.imap())')
    else:
        raise AssertionError('expected ZeroDivisionError')

    it = pool.imap(f, range(10))
    for i in range(10):
        try:
            x = it.next()
        except ZeroDivisionError:
            if i == 5:
                pass
        except StopIteration:
            break
        else:
            if i == 5:
                raise AssertionError('expected ZeroDivisionError')
            
    assert i == 9
    print('\tGot ZeroDivisionError as expected from IMapIterator.next()')
    print()
    
    #
    # Testing timeouts
    #
    
    print('Testing ApplyResult.get() with timeout:', end='')
    res = pool.apply_async(calculate, TASKS[0])
    while 1:
        sys.stdout.flush()
        try:
            sys.stdout.write('\n\t%s' % res.get(0.02))
            break
        except TimeoutError:
            sys.stdout.write('.')
    print()
    print()

    print('Testing IMapIterator.next() with timeout:', end='')
    it = pool.imap(calculatestar, TASKS)
    while 1:
        sys.stdout.flush()
        try:
            sys.stdout.write('\n\t%s' % it.next(0.02))
        except StopIteration:
            break
        except TimeoutError:
            sys.stdout.write('.')
    print()
    print()
            
    #
    # Testing callback
    #

    print('Testing callback:')
    
    A = []
    B = [56, 0, 1, 8, 27, 64, 125, 216, 343, 512, 729]
        
    r = pool.apply_async(mul, (7, 8), callback=A.append)
    r.wait()

    r = pool.map_async(pow3, range(10), callback=A.extend)
    r.wait()

    if A == B:
        print('\tcallbacks succeeded\n')
    else:
        print('\t*** callbacks failed\n\t\t%s != %s\n' % (A, B))
    
    #
    # Check there are no outstanding tasks
    #
    
    assert not pool._cache, 'cache = %r' % pool._cache

    #
    # Check close() methods
    #

    print('Testing close():')

    for worker in pool._pool:
        assert worker.is_alive()

    result = pool.apply_async(time.sleep, [0.5])
    pool.close()
    pool.join()

    assert result.get() is None

    for worker in pool._pool:
        assert not worker.is_alive()

    print('\tclose() succeeded\n')

    #
    # Check terminate() method
    #

    print('Testing terminate():')

    pool = Pool(2)
    ignore = pool.apply(pow3, [2])
    results = [pool.apply_async(time.sleep, [10]) for i in range(10)]
    pool.terminate()
    pool.join()

    for worker in pool._pool:
        assert not worker.is_alive()

    print('\tterminate() succeeded\n')

    #
    # Check garbage collection
    #

    print('Testing garbage collection:')

    pool = Pool(2)
    processes = pool._pool
    
    ignore = pool.apply(pow3, [2])
    results = [pool.apply_async(time.sleep, [10]) for i in range(10)]

    del results, pool

    time.sleep(0.2)
    
    for worker in processes:
        assert not worker.is_alive()

    print('\tgarbage collection succeeded\n')

Example 145

Project: clusterd
Source File: authenticate.py
View license
def checkAuth(ip, port, title, version):
    """
    """

    if version in ["5.1", "6.0", "6.1"] and title is JINTERFACES.WM:
        for (usr, pswd) in default_credentials:
            url = "http://%s:%s/admin-console/login.seam" % (ip, port)
            data = OrderedDict([
                    ("login_form", "login_form"),
                    ("login_form:name", usr),
                    ("login_form:password", pswd),
                    ("login_form:submit", "Login"),
                    ("javax.faces.ViewState", utility.fetch_viewState(url)),
                   ])

            response = utility.requests_post(url, data=data)
            if response.status_code == 200:
                utility.Msg("Successfully authenticated with %s:%s" % (usr, pswd), LOG.DEBUG)
                if version in ["5.1"]:
                    return (dict_from_cookiejar(response.history[0].cookies), None)
                return (dict_from_cookiejar(response.cookies), None)

    else:
        if title is JINTERFACES.JMX:
            url = "http://%s:%s/jmx-console/" % (ip, port)
        elif title is JINTERFACES.MM:
            url = "http://%s:%s/management" % (ip, port)
        elif title is JINTERFACES.WC:
            url = "http://%s:%s/web-console" % (ip, port)
        else:
            utility.Msg("Unsupported auth interface: %s" % title, LOG.DEBUG)
            return

        # check with given auth
        if state.usr_auth:
            (usr, pswd) = state.usr_auth.split(':')
            return _auth(usr, pswd, url, version)

        # else try default credentials
        for (usr, pswd) in default_credentials:
            cook = _auth(usr, pswd, url, version)
            if cook:
                return cook

        # if we're still here, check if they supplied a wordlist
        if state.bf_wordlist and not state.hasbf:

            state.hasbf = True
            wordlist = []
            with open(state.bf_wordlist, 'r') as f:
                # ensure everything is ascii or requests will explode
                wordlist = [x.decode("ascii", "ignore").rstrip() for x in f.readlines()]

            utility.Msg("Brute forcing %s account with %d passwords..." %
                                        (state.bf_user, len(wordlist)), LOG.DEBUG)

            try:
                for (idx, word) in enumerate(wordlist):
                    stdout.flush()
                    stdout.write("\r\033[32m [%s] Brute forcing password for %s [%d/%d]\033[0m" \
                                        % (utility.timestamp(), state.bf_user,
                                           idx+1, len(wordlist)))

                    cook = _auth(state.bf_user, word, url, version)
                    if cook:
                        print ''  # newline

                        # lets insert these credentials to the default list so we
                        # don't need to bruteforce it each time
                        if not (state.bf_user, word) in default_credentials:
                            default_credentials.insert(0, (state.bf_user, word))

                        utility.Msg("Successful login %s:%s" % 
                                        (state.bf_user, word), LOG.SUCCESS)
                        return cook

                print ''

            except KeyboardInterrupt:
                pass

Example 146

View license
def main_train():
    """Step 1 : Download Training and Testing data.
    Compare with Word2vec example, the dataset in this example is large,
    so we use TensorFlow's gfile functions to speed up the pre-processing.
    """
    print()
    print("Prepare raw data")
    train_path, dev_path = tl.files.load_wmt_en_fr_dataset(data_dir=data_dir)
    print("Training data : %s" % train_path)   # wmt/giga-fren.release2
    print("Testing data : %s" % dev_path)     # wmt/newstest2013

    """Step 2 : Create Vocabularies for both Training and Testing data.
    """
    print()
    print("Create vocabularies")
    fr_vocab_path = os.path.join(data_dir, "vocab%d.fr" % fr_vocab_size)
    en_vocab_path = os.path.join(data_dir, "vocab%d.en" % en_vocab_size)
    print("Vocabulary of French : %s" % fr_vocab_path)    # wmt/vocab40000.fr
    print("Vocabulary of English : %s" % en_vocab_path)   # wmt/vocab40000.en
    tl.nlp.create_vocabulary(fr_vocab_path, train_path + ".fr",
                fr_vocab_size, tokenizer=None, normalize_digits=normalize_digits,
                _DIGIT_RE=_DIGIT_RE, _START_VOCAB=_START_VOCAB)
    tl.nlp.create_vocabulary(en_vocab_path, train_path + ".en",
                en_vocab_size, tokenizer=None, normalize_digits=normalize_digits,
                _DIGIT_RE=_DIGIT_RE, _START_VOCAB=_START_VOCAB)

    """ Step 3 : Tokenize Training and Testing data.
    """
    print()
    print("Tokenize data")
    # Create tokenized file for the training data by using the vocabulary file.
    # normalize_digits=True means set all digits to zero, so as to reduce
    # vocabulary size.
    fr_train_ids_path = train_path + (".ids%d.fr" % fr_vocab_size)
    en_train_ids_path = train_path + (".ids%d.en" % en_vocab_size)
    print("Tokenized Training data of French : %s" % fr_train_ids_path)    # wmt/giga-fren.release2.ids40000.fr
    print("Tokenized Training data of English : %s" % en_train_ids_path)   # wmt/giga-fren.release2.ids40000.fr
    tl.nlp.data_to_token_ids(train_path + ".fr", fr_train_ids_path, fr_vocab_path,
                                tokenizer=None, normalize_digits=normalize_digits,
                                UNK_ID=UNK_ID, _DIGIT_RE=_DIGIT_RE)
    tl.nlp.data_to_token_ids(train_path + ".en", en_train_ids_path, en_vocab_path,
                                tokenizer=None, normalize_digits=normalize_digits,
                                UNK_ID=UNK_ID, _DIGIT_RE=_DIGIT_RE)

    # we should also create tokenized file for the development (testing) data.
    fr_dev_ids_path = dev_path + (".ids%d.fr" % fr_vocab_size)
    en_dev_ids_path = dev_path + (".ids%d.en" % en_vocab_size)
    print("Tokenized Testing data of French : %s" % fr_dev_ids_path)    # wmt/newstest2013.ids40000.fr
    print("Tokenized Testing data of English : %s" % en_dev_ids_path)   # wmt/newstest2013.ids40000.en
    tl.nlp.data_to_token_ids(dev_path + ".fr", fr_dev_ids_path, fr_vocab_path,
                                tokenizer=None, normalize_digits=normalize_digits,
                                UNK_ID=UNK_ID, _DIGIT_RE=_DIGIT_RE)
    tl.nlp.data_to_token_ids(dev_path + ".en", en_dev_ids_path, en_vocab_path,
                                tokenizer=None, normalize_digits=normalize_digits,
                                UNK_ID=UNK_ID, _DIGIT_RE=_DIGIT_RE)

    # You can get the word_to_id dictionary and id_to_word list as follow.
    # vocab, rev_vocab = tl.nlp.initialize_vocabulary(en_vocab_path)
    # print(vocab)
    # {b'cat': 1, b'dog': 0, b'bird': 2}
    # print(rev_vocab)
    # [b'dog', b'cat', b'bird']

    en_train = en_train_ids_path
    fr_train = fr_train_ids_path
    en_dev = en_dev_ids_path
    fr_dev = fr_dev_ids_path

    """Step 4 : Load both tokenized Training and Testing data into buckets
    and compute their size.

    Bucketing is a method to efficiently handle sentences of different length.
    When translating English to French, we will have English sentences of
    different lengths I on input, and French sentences of different
    lengths O on output. We should in principle create a seq2seq model
    for every pair (I, O+1) of lengths of an English and French sentence.

    For find the closest bucket for each pair, then we could just pad every
    sentence with a special PAD symbol in the end if the bucket is bigger
    than the sentence

    We use a number of buckets and pad to the closest one for efficiency.

    If the input is an English sentence with 3 tokens, and the corresponding
    output is a French sentence with 6 tokens, then they will be put in the
    first bucket and padded to length 5 for encoder inputs (English sentence),
    and length 10 for decoder inputs.
    If we have an English sentence with 8 tokens and the corresponding French
    sentence has 18 tokens, then they will be fit into (20, 25) bucket.

    Given a pair [["I", "go", "."], ["Je", "vais", "."]] in tokenized format.
    The training data of encoder inputs representing [PAD PAD "." "go" "I"]
    and decoder inputs [GO "Je" "vais" "." EOS PAD PAD PAD PAD PAD].
    see ``get_batch()``
    """
    print()
    print ("Read development (test) data into buckets")
    dev_set = read_data(en_dev, fr_dev, buckets, EOS_ID)

    if plot_data:
        # Visualize the development (testing) data
        print('dev data:', buckets[0], dev_set[0][0])    # (5, 10), [[13388, 4, 949], [23113, 8, 910, 2]]
        vocab_en, rev_vocab_en = tl.nlp.initialize_vocabulary(en_vocab_path)
        context = tl.nlp.word_ids_to_words(dev_set[0][0][0], rev_vocab_en)
        word_ids = tl.nlp.words_to_word_ids(context, vocab_en)
        print('en word_ids:', word_ids) # [13388, 4, 949]
        print('en context:', context)   # [b'Preventing', b'the', b'disease']
        vocab_fr, rev_vocab_fr = tl.nlp.initialize_vocabulary(fr_vocab_path)
        context = tl.nlp.word_ids_to_words(dev_set[0][0][1], rev_vocab_fr)
        word_ids = tl.nlp.words_to_word_ids(context, vocab_fr)
        print('fr word_ids:', word_ids) # [23113, 8, 910, 2]
        print('fr context:', context)   # [b'Pr\xc3\xa9venir', b'la', b'maladie', b'_EOS']

    print()
    print ("Read training data into buckets (limit: %d)" % max_train_data_size)
    train_set = read_data(en_train, fr_train, buckets, EOS_ID, max_train_data_size)
    if plot_data:
        # Visualize the training data
        print('train data:', buckets[0], train_set[0][0])   # (5, 10) [[1368, 3344], [1089, 14, 261, 2]]
        context = tl.nlp.word_ids_to_words(train_set[0][0][0], rev_vocab_en)
        word_ids = tl.nlp.words_to_word_ids(context, vocab_en)
        print('en word_ids:', word_ids) # [1368, 3344]
        print('en context:', context)   # [b'Site', b'map']
        context = tl.nlp.word_ids_to_words(train_set[0][0][1], rev_vocab_fr)
        word_ids = tl.nlp.words_to_word_ids(context, vocab_fr)
        print('fr word_ids:', word_ids) # [1089, 14, 261, 2]
        print('fr context:', context)   # [b'Plan', b'du', b'site', b'_EOS']
        print()

    train_bucket_sizes = [len(train_set[b]) for b in xrange(len(buckets))]
    train_total_size = float(sum(train_bucket_sizes))
    print('the num of training data in each buckets: %s' % train_bucket_sizes)    # [239121, 1344322, 5239557, 10445326]
    print('the num of training data: %d' % train_total_size)        # 17268326.0

    # A bucket scale is a list of increasing numbers from 0 to 1 that we'll use
    # to select a bucket. Length of [scale[i], scale[i+1]] is proportional to
    # the size if i-th training bucket, as used later.
    train_buckets_scale = [sum(train_bucket_sizes[:i + 1]) / train_total_size
                           for i in xrange(len(train_bucket_sizes))]
    print('train_buckets_scale:',train_buckets_scale)   # [0.013847375825543252, 0.09169638099257565, 0.3951164693091849, 1.0]


    """Step 6 : Create model
    """
    print()
    print("Create Embedding Attention Seq2seq Model")
    with tf.variable_scope("model", reuse=None):
        model = tl.layers.EmbeddingAttentionSeq2seqWrapper(
                          en_vocab_size,
                          fr_vocab_size,
                          buckets,
                          size,
                          num_layers,
                          max_gradient_norm,
                          batch_size,
                          learning_rate,
                          learning_rate_decay_factor,
                          forward_only=False)    # is_train = True

    sess.run(tf.initialize_all_variables())
    # model.print_params()
    tl.layers.print_all_variables()

    if resume:
        print("Load existing model" + "!"*10)
        if is_npz:
            # instead of using TensorFlow saver, we can use TensorLayer to restore a model
            load_params = tl.files.load_npz(name=model_file_name+'.npz')
            tl.files.assign_params(sess, load_params, model)
        else:
            saver = tf.train.Saver()
            saver.restore(sess, model_file_name+'.ckpt')

    """Step 7 : Training
    """
    print()
    step_time, loss = 0.0, 0.0
    current_step = 0
    previous_losses = []
    while True:
        # Choose a bucket according to data distribution. We pick a random number
        # in [0, 1] and use the corresponding interval in train_buckets_scale.
        random_number_01 = np.random.random_sample()
        bucket_id = min([i for i in xrange(len(train_buckets_scale))
                       if train_buckets_scale[i] > random_number_01])

        # Get a batch and make a step.
        # randomly pick ``batch_size`` training examples from a random bucket_id
        # the data format is described in readthedocs tutorial
        start_time = time.time()
        encoder_inputs, decoder_inputs, target_weights = model.get_batch(
                train_set, bucket_id, PAD_ID, GO_ID, EOS_ID, UNK_ID)

        _, step_loss, _ = model.step(sess, encoder_inputs, decoder_inputs,
                                   target_weights, bucket_id, False)
        step_time += (time.time() - start_time) / steps_per_checkpoint
        loss += step_loss / steps_per_checkpoint
        current_step += 1

        # Once in a while, we save checkpoint, print statistics, and run evals.
        if current_step % steps_per_checkpoint == 0:
            # Print statistics for the previous epoch.
            perplexity = math.exp(loss) if loss < 300 else float('inf')
            print ("global step %d learning rate %.4f step-time %.2f perplexity "
                "%.2f" % (model.global_step.eval(), model.learning_rate.eval(),
                            step_time, perplexity))
            # Decrease learning rate if no improvement was seen over last 3 times.
            if len(previous_losses) > 2 and loss > max(previous_losses[-3:]):
                sess.run(model.learning_rate_decay_op)
            previous_losses.append(loss)

            # Save model
            if is_npz:
                tl.files.save_npz(model.all_params, name=model_file_name+'.npz')
            else:
                print('Model is saved to: %s' % model_file_name+'.ckpt')
                checkpoint_path = os.path.join(train_dir, model_file_name+'.ckpt')
                model.saver.save(sess, checkpoint_path, global_step=model.global_step)

            step_time, loss = 0.0, 0.0
            # Run evals on development set and print their perplexity.
            for bucket_id in xrange(len(buckets)):
                if len(dev_set[bucket_id]) == 0:
                    print("  eval: empty bucket %d" % (bucket_id))
                    continue
                encoder_inputs, decoder_inputs, target_weights = model.get_batch(
                        dev_set, bucket_id, PAD_ID, GO_ID, EOS_ID, UNK_ID)
                _, eval_loss, _ = model.step(sess, encoder_inputs, decoder_inputs,
                                               target_weights, bucket_id, True)
                eval_ppx = math.exp(eval_loss) if eval_loss < 300 else float('inf')
                print("  eval: bucket %d perplexity %.2f" % (bucket_id, eval_ppx))
            sys.stdout.flush()

Example 147

Project: multiprocess
Source File: ex_pool.py
View license
def test():
    print('cpuCount() = %d\n' % cpuCount())
    
    #
    # Create pool
    #
    
    PROCESSES = 4
    print('Creating pool with %d processes\n' % PROCESSES)
    pool = Pool(PROCESSES)    

    #
    # Tests
    #

    TASKS = [(mul, (i, 7)) for i in range(10)] + \
            [(plus, (i, 8)) for i in range(10)]

    results = [pool.apply_async(calculate, t) for t in TASKS]
    imap_it = pool.imap(calculatestar, TASKS)
    imap_unordered_it = pool.imap_unordered(calculatestar, TASKS)

    print('Ordered results using pool.apply_async():')
    for r in results:
        print('\t', r.get())
    print()

    print('Ordered results using pool.imap():')
    for x in imap_it:
        print('\t', x)
    print()

    print('Unordered results using pool.imap_unordered():')
    for x in imap_unordered_it:
        print('\t', x)
    print()

    print('Ordered results using pool.map() --- will block till complete:')
    for x in pool.map(calculatestar, TASKS):
        print('\t', x)
    print()

    #
    # Simple benchmarks
    #

    N = 100000
    print('def pow3(x): return x**3')
    
    t = time.time()
    A = list(map(pow3, xrange(N)))
    print('\tmap(pow3, xrange(%d)):\n\t\t%s seconds' % \
          (N, time.time() - t))
    
    t = time.time()
    B = pool.map(pow3, xrange(N))
    print('\tpool.map(pow3, xrange(%d)):\n\t\t%s seconds' % \
          (N, time.time() - t))

    t = time.time()
    C = list(pool.imap(pow3, xrange(N), chunksize=N//8))
    print('\tlist(pool.imap(pow3, xrange(%d), chunksize=%d)):\n\t\t%s' \
          ' seconds' % (N, N//8, time.time() - t))
    
    assert A == B == C, (len(A), len(B), len(C))
    print()
    
    L = [None] * 1000000
    print('def noop(x): pass')
    print('L = [None] * 1000000')
    
    t = time.time()
    A = list(map(noop, L))
    print('\tmap(noop, L):\n\t\t%s seconds' % \
          (time.time() - t))
    
    t = time.time()
    B = pool.map(noop, L)
    print('\tpool.map(noop, L):\n\t\t%s seconds' % \
          (time.time() - t))

    t = time.time()
    C = list(pool.imap(noop, L, chunksize=len(L)//8))
    print('\tlist(pool.imap(noop, L, chunksize=%d)):\n\t\t%s seconds' % \
          (len(L)//8, time.time() - t))

    assert A == B == C, (len(A), len(B), len(C))
    print()

    del A, B, C, L

    #
    # Test error handling
    #

    print('Testing error handling:')

    try:
        print(pool.apply(f, (5,)))
    except ZeroDivisionError:
        print('\tGot ZeroDivisionError as expected from pool.apply()')
    else:
        raise AssertionError('expected ZeroDivisionError')

    try:
        print(pool.map(f, range(10)))
    except ZeroDivisionError:
        print('\tGot ZeroDivisionError as expected from pool.map()')
    else:
        raise AssertionError('expected ZeroDivisionError')
            
    try:
        print(list(pool.imap(f, range(10))))
    except ZeroDivisionError:
        print('\tGot ZeroDivisionError as expected from list(pool.imap())')
    else:
        raise AssertionError('expected ZeroDivisionError')

    it = pool.imap(f, range(10))
    for i in range(10):
        try:
            x = it.next()
        except ZeroDivisionError:
            if i == 5:
                pass
        except StopIteration:
            break
        else:
            if i == 5:
                raise AssertionError('expected ZeroDivisionError')
            
    assert i == 9
    print('\tGot ZeroDivisionError as expected from IMapIterator.next()')
    print()
    
    #
    # Testing timeouts
    #
    
    print('Testing ApplyResult.get() with timeout:', end='')
    res = pool.apply_async(calculate, TASKS[0])
    while 1:
        sys.stdout.flush()
        try:
            sys.stdout.write('\n\t%s' % res.get(0.02))
            break
        except TimeoutError:
            sys.stdout.write('.')
    print()
    print()

    print('Testing IMapIterator.next() with timeout:', end='')
    it = pool.imap(calculatestar, TASKS)
    while 1:
        sys.stdout.flush()
        try:
            sys.stdout.write('\n\t%s' % it.next(0.02))
        except StopIteration:
            break
        except TimeoutError:
            sys.stdout.write('.')
    print()
    print()
            
    #
    # Testing callback
    #

    print('Testing callback:')
    
    A = []
    B = [56, 0, 1, 8, 27, 64, 125, 216, 343, 512, 729]
        
    r = pool.apply_async(mul, (7, 8), callback=A.append)
    r.wait()

    r = pool.map_async(pow3, range(10), callback=A.extend)
    r.wait()

    if A == B:
        print('\tcallbacks succeeded\n')
    else:
        print('\t*** callbacks failed\n\t\t%s != %s\n' % (A, B))
    
    #
    # Check there are no outstanding tasks
    #
    
    assert not pool._cache, 'cache = %r' % pool._cache

    #
    # Check close() methods
    #

    print('Testing close():')

    for worker in pool._pool:
        assert worker.is_alive()

    result = pool.apply_async(time.sleep, [0.5])
    pool.close()
    pool.join()

    assert result.get() is None

    for worker in pool._pool:
        assert not worker.is_alive()

    print('\tclose() succeeded\n')

    #
    # Check terminate() method
    #

    print('Testing terminate():')

    pool = Pool(2)
    ignore = pool.apply(pow3, [2])
    results = [pool.apply_async(time.sleep, [10]) for i in range(10)]
    pool.terminate()
    pool.join()

    for worker in pool._pool:
        assert not worker.is_alive()

    print('\tterminate() succeeded\n')

    #
    # Check garbage collection
    #

    print('Testing garbage collection:')

    pool = Pool(2)
    processes = pool._pool
    
    ignore = pool.apply(pow3, [2])
    results = [pool.apply_async(time.sleep, [10]) for i in range(10)]

    del results, pool

    time.sleep(0.2)
    
    for worker in processes:
        assert not worker.is_alive()

    print('\tgarbage collection succeeded\n')

Example 148

Project: ikaaro
Source File: icms-update-catalog.py
View license
def update_catalog(parser, options, target):
    # Check the server is not started, or started in read-only mode
    server = Server(target, read_only=True, cache_size=options.cache_size)
    if server.is_running_in_rw_mode():
        print 'Cannot proceed, the server is running in read-write mode.'
        return

    # Check for database consistency
    if options.quick is False and check_database(target) is False:
        return 1

    # Ask
    message = 'Update the catalog (y/N)? '
    if ask_confirmation(message, options.confirm) is False:
        return

    # Create a temporary new catalog
    catalog_path = '%s/catalog.new' % target
    if lfs.exists(catalog_path):
        lfs.remove(catalog_path)
    catalog = make_catalog(catalog_path, get_register_fields())

    # Get the root
    root = server.root

    # Build a fake context
    context = get_fake_context(server.database)
    context.server = server

    # Update
    t0, v0 = time(), vmsize()
    doc_n = 0
    error_detected = False
    if options.test:
        log = open('%s/log/update-catalog' % target, 'w').write
    for obj in root.traverse_resources():
        if not isinstance(obj, Resource):
            continue
        if not options.quiet:
            print doc_n, obj.abspath
        doc_n += 1
        context.resource = obj

        # Index the document
        try:
            catalog.index_document(obj)
        except Exception:
            if options.test:
                error_detected = True
                log('*** Error detected ***\n')
                log('Abspath of the resource: %r\n\n' % str(obj.abspath))
                log(format_exc())
                log('\n')
            else:
                raise

        # Free Memory
        del obj
        server.database.make_room()

    if not error_detected:
        if options.test:
            # Delete the empty log file
            remove('%s/log/update-catalog' % target)

        # Update / Report
        t1, v1 = time(), vmsize()
        v = (v1 - v0)/1024
        print '[Update] Time: %.02f seconds. Memory: %s Kb' % (t1 - t0, v)

        # Commit
        print '[Commit]',
        sys.stdout.flush()
        catalog.save_changes()
        # Commit / Replace
        old_catalog_path = '%s/catalog' % target
        if lfs.exists(old_catalog_path):
            lfs.remove(old_catalog_path)
        lfs.move(catalog_path, old_catalog_path)
        # Commit / Report
        t2, v2 = time(), vmsize()
        v = (v2 - v1)/1024
        print 'Time: %.02f seconds. Memory: %s Kb' % (t2 - t1, v)
    else:
        print '[Update] Error(s) detected, the new catalog was NOT saved'
        print ('[Update] You can find more infos in %r' %
               join(target, 'log/update-catalog'))

Example 149

Project: droidbot
Source File: droidbox.py
View license
    def start_blocked(self, duration=0):
        if not self.enabled:
            return
        # curses.setupterm()
        # sys.stdout.write(curses.tigetstr("clear"))
        sys.stdout.flush()
        call(["adb", "wait-for-device"])
        call(['adb', 'logcat', '-c'])

        print " ____                        __  ____"
        print "/\  _`\               __    /\ \/\  _`\\"
        print "\ \ \/\ \  _ __  ___ /\_\   \_\ \ \ \L\ \   ___   __  _"
        print " \ \ \ \ \/\`'__\ __`\/\ \  /'_` \ \  _ <' / __`\/\ \/'\\"
        print "  \ \ \_\ \ \ \/\ \L\ \ \ \/\ \L\ \ \ \L\ \\ \L\ \/>  </"
        print "   \ \____/\ \_\ \____/\ \_\ \___,_\ \____/ \____//\_/\_\\"
        print "    \/___/  \/_/\/___/  \/_/\/__,_ /\/___/ \/___/ \//\/_/"

        counter = CountingThread()
        counter.start()

        if duration:
            self.timer = threading.Timer(duration, self.stop)
            self.timer.start()

        if self.logcat is None:
            self.logcat = Popen(["adb", "logcat", "-v", "threadtime", "DroidBox:W", "dalvikvm:W", "ActivityManager:I"],
                                stdin=subprocess.PIPE, stdout=subprocess.PIPE)

        # Collect DroidBox logs
        self.is_counting_logs = True
        self.lastScreenshot = 0
        first_log_time = None

        fd2path = {}

        while self.enabled:
            try:
                if self.output_dir and (time.time() - self.lastScreenshot) >= 5:
                    # Take screenshots every 5 seconds.
                    os.system("adb shell screencap -p | sed 's/\r$//' > %s" % os.path.join(self.output_dir, "screen") \
                              + "_$(date +%Y-%m-%d_%H%M%S).png")
                    self.lastScreenshot = time.time()

                logcatInput = self.logcat.stdout.readline()
                if not logcatInput:
                    raise LostADBException("We have lost the connection with ADB.")

                log_data = parse_log(logcatInput)

                if log_data is None or not log_data['content'].startswith("DroidBox:"):
                    continue

                log_time = log_data['datetime']
                if first_log_time is None:
                    first_log_time = log_time
                log_delta_seconds = (log_time - first_log_time).total_seconds()

                log_content = json.loads(decode(log_data['content'][10:]))

                log_process_names = self.state_monitor.get_names_by_pid(log_data['pid'])
                log_process_name = "->".join(log_process_names)

                for log_type in log_content:
                    log_detail = log_content[log_type]
                    if log_type == "FdAccess":
                        path = hexToStr(log_detail['path'])
                        fd2path[log_detail['id']] = path
                        log_detail['path'] = path
                    if log_type == "FileRW" and log_detail['id'] in fd2path:
                        log_detail['path'] = fd2path[log_detail['id']]
                    if log_type == "DataLeak":
                        log_detail['tag'] = getTags(int(log_detail['tag'], 16))
                        if log_detail['sink'] == "File" and log_detail['id'] in fd2path:
                            log_detail['path'] = fd2path[log_detail['id']]

                    log_dict = {"type": log_type,
                                "time": log_delta_seconds,
                                "process": log_process_name,
                                "detail": log_detail}

                    if self.filter_noises(log_dict):
                        continue

                    self.sensitive_behaviors.append(log_dict)
                    counter.increaseCount()
            except KeyboardInterrupt:
                break
            except LostADBException:
                break
            except Exception as e:
                print(e.message)
                continue

        self.is_counting_logs = False
        counter.stopCounting()
        counter.join()
        # Kill ADB, otherwise it will never terminate
        self.stop()
        self.logcat = None

        print json.dumps(self.get_output())
        if self.output_dir is None:
            return
        with open(os.path.join(self.output_dir, "analysis.json"), "w") as json_file:
            json_file.write(json.dumps(self.get_output(), sort_keys=True, indent=4))

Example 150

Project: multiprocess
Source File: ex_pool.py
View license
def test():
    print('cpuCount() = %d\n' % cpuCount())
    
    #
    # Create pool
    #
    
    PROCESSES = 4
    print('Creating pool with %d processes\n' % PROCESSES)
    pool = Pool(PROCESSES)    

    #
    # Tests
    #

    TASKS = [(mul, (i, 7)) for i in range(10)] + \
            [(plus, (i, 8)) for i in range(10)]

    results = [pool.apply_async(calculate, t) for t in TASKS]
    imap_it = pool.imap(calculatestar, TASKS)
    imap_unordered_it = pool.imap_unordered(calculatestar, TASKS)

    print('Ordered results using pool.apply_async():')
    for r in results:
        print('\t', r.get())
    print()

    print('Ordered results using pool.imap():')
    for x in imap_it:
        print('\t', x)
    print()

    print('Unordered results using pool.imap_unordered():')
    for x in imap_unordered_it:
        print('\t', x)
    print()

    print('Ordered results using pool.map() --- will block till complete:')
    for x in pool.map(calculatestar, TASKS):
        print('\t', x)
    print()

    #
    # Simple benchmarks
    #

    N = 100000
    print('def pow3(x): return x**3')
    
    t = time.time()
    A = list(map(pow3, xrange(N)))
    print('\tmap(pow3, xrange(%d)):\n\t\t%s seconds' % \
          (N, time.time() - t))
    
    t = time.time()
    B = pool.map(pow3, xrange(N))
    print('\tpool.map(pow3, xrange(%d)):\n\t\t%s seconds' % \
          (N, time.time() - t))

    t = time.time()
    C = list(pool.imap(pow3, xrange(N), chunksize=N//8))
    print('\tlist(pool.imap(pow3, xrange(%d), chunksize=%d)):\n\t\t%s' \
          ' seconds' % (N, N//8, time.time() - t))
    
    assert A == B == C, (len(A), len(B), len(C))
    print()
    
    L = [None] * 1000000
    print('def noop(x): pass')
    print('L = [None] * 1000000')
    
    t = time.time()
    A = list(map(noop, L))
    print('\tmap(noop, L):\n\t\t%s seconds' % \
          (time.time() - t))
    
    t = time.time()
    B = pool.map(noop, L)
    print('\tpool.map(noop, L):\n\t\t%s seconds' % \
          (time.time() - t))

    t = time.time()
    C = list(pool.imap(noop, L, chunksize=len(L)//8))
    print('\tlist(pool.imap(noop, L, chunksize=%d)):\n\t\t%s seconds' % \
          (len(L)//8, time.time() - t))

    assert A == B == C, (len(A), len(B), len(C))
    print()

    del A, B, C, L

    #
    # Test error handling
    #

    print('Testing error handling:')

    try:
        print(pool.apply(f, (5,)))
    except ZeroDivisionError:
        print('\tGot ZeroDivisionError as expected from pool.apply()')
    else:
        raise AssertionError('expected ZeroDivisionError')

    try:
        print(pool.map(f, range(10)))
    except ZeroDivisionError:
        print('\tGot ZeroDivisionError as expected from pool.map()')
    else:
        raise AssertionError('expected ZeroDivisionError')
            
    try:
        print(list(pool.imap(f, range(10))))
    except ZeroDivisionError:
        print('\tGot ZeroDivisionError as expected from list(pool.imap())')
    else:
        raise AssertionError('expected ZeroDivisionError')

    it = pool.imap(f, range(10))
    for i in range(10):
        try:
            x = it.next()
        except ZeroDivisionError:
            if i == 5:
                pass
        except StopIteration:
            break
        else:
            if i == 5:
                raise AssertionError('expected ZeroDivisionError')
            
    assert i == 9
    print('\tGot ZeroDivisionError as expected from IMapIterator.next()')
    print()
    
    #
    # Testing timeouts
    #
    
    print('Testing ApplyResult.get() with timeout:', end='')
    res = pool.apply_async(calculate, TASKS[0])
    while 1:
        sys.stdout.flush()
        try:
            sys.stdout.write('\n\t%s' % res.get(0.02))
            break
        except TimeoutError:
            sys.stdout.write('.')
    print()
    print()

    print('Testing IMapIterator.next() with timeout:', end='')
    it = pool.imap(calculatestar, TASKS)
    while 1:
        sys.stdout.flush()
        try:
            sys.stdout.write('\n\t%s' % it.next(0.02))
        except StopIteration:
            break
        except TimeoutError:
            sys.stdout.write('.')
    print()
    print()
            
    #
    # Testing callback
    #

    print('Testing callback:')
    
    A = []
    B = [56, 0, 1, 8, 27, 64, 125, 216, 343, 512, 729]
        
    r = pool.apply_async(mul, (7, 8), callback=A.append)
    r.wait()

    r = pool.map_async(pow3, range(10), callback=A.extend)
    r.wait()

    if A == B:
        print('\tcallbacks succeeded\n')
    else:
        print('\t*** callbacks failed\n\t\t%s != %s\n' % (A, B))
    
    #
    # Check there are no outstanding tasks
    #
    
    assert not pool._cache, 'cache = %r' % pool._cache

    #
    # Check close() methods
    #

    print('Testing close():')

    for worker in pool._pool:
        assert worker.is_alive()

    result = pool.apply_async(time.sleep, [0.5])
    pool.close()
    pool.join()

    assert result.get() is None

    for worker in pool._pool:
        assert not worker.is_alive()

    print('\tclose() succeeded\n')

    #
    # Check terminate() method
    #

    print('Testing terminate():')

    pool = Pool(2)
    ignore = pool.apply(pow3, [2])
    results = [pool.apply_async(time.sleep, [10]) for i in range(10)]
    pool.terminate()
    pool.join()

    for worker in pool._pool:
        assert not worker.is_alive()

    print('\tterminate() succeeded\n')

    #
    # Check garbage collection
    #

    print('Testing garbage collection:')

    pool = Pool(2)
    processes = pool._pool
    
    ignore = pool.apply(pow3, [2])
    results = [pool.apply_async(time.sleep, [10]) for i in range(10)]

    del results, pool

    time.sleep(0.2)
    
    for worker in processes:
        assert not worker.is_alive()

    print('\tgarbage collection succeeded\n')