Softpanorama

May the source be with you, but remember the KISS principle ;-)
Contents Bulletin Scripting in shell and Perl Network troubleshooting History Humor

Python -- Scripting language with generators and coroutines

News Scripting Languages

Best Python books for system administrators

Recommended Links Python Braces Debate Command-Line Syntax and Options pdb — The Python Debugger
Tutorials Coroutines Braces Problem Programming environment Python IDEs Pycharm IDE Jython
             
Debugging in Python Algorithms Quotes Python history Tips Etc  
 
A new competitor seemed to emerge out of the woodwork every month or so. The first thing I would do, after checking to see if they had a live online demo, was look at their job listings. After a couple years of this I could tell which companies to worry about and which not to. The more of an IT flavor the job descriptions had, the less dangerous the company was.
  • The safest kind were the ones that wanted Oracle experience. You never had to worry about those.
    You were also safe if they said they wanted C++ or Java developers.
  • If they wanted Perl or Python programmers, that would be a bit frightening-- that's starting to sound like a company where the technical side, at least, is run by real hackers.
  • If I had ever seen a job posting looking for Lisp hackers, I would have been really worried.

-- Paul Graham co-founder, Viaweb

This is the forth page of an ongoing series of pages covering scripting language topics for the Unix/Linux system administrators  (others cover Unix shells, Perl, and  TCL

Python now is becoming the language Unix/linux sysadmin must know at least on superficial level as for many users it is the primary language iether for development or for writing supporting scripts. Python has been hailed as a really easy to learn language. It's not completely true, but it get a strong foothold at universities (repleating path of Pascal in this regard, but on  a new level). The rest is history.

As most sysadmins know Perl, Python should be easy to learn as key concepts are similar and Python is the language influenced by Perl, incorporating some European and, more specifically, Niklaus Wirth ideas about programming languages. Python's core syntax and certain aspects of its philosophy were also influenced from ABC.  But it is not true. It's pretty difficult and even annoying to learn for accomplished Perl programmers. More about this at

Python programs can be decomposed into modules, statements, expressions, and objects, as follows:

  1. Programs are composed of modules.
  2. Modules contain statements.
  3. Statements contain expressions.
  4. Expressions create and process objects.

First versions of Python did not introduced any new ideas and by-and large was just more cleaner rehash of ideas Perl with the addition of idea of modules from Module 3.  It was released in 1991, the same year as Linux. Wikipedia has a short article about Python history:  History of Python - Wikipedia, the free encyclopedia

But starting with version 2.2 it added support of semi-coroutines in a platform independent way (via generators, the concept inspired by Icon) and became class of its own.  I think that this makes Python in certain areas a better scripting language then other members of the "most popular troika" (Perl, PHP and JavaScript). Availability of semi-coroutines favorably distinguish Python from Perl.

Python was lucky that for some time it enjoyed full support of Google (which still employs Python creator,  Guido van Rossum). In addition Microsoft also supported it indirectly (via Iron Python and support In Visual Studio and other tools like Expressions Web).  That created some money flow for development and put Python in much better position then Perl, which after it lost O'Reilly does not have powerful corporate sponsor and this development is lingering in obscurity.

Even in 2017 Python still enjoy some level of support of Google, and that are no similar sponsor for iether Ruby or R, or Perl.  Out of modern languages only  JavaScript survived a semi-abandoned status (after collapse of Netscape),  but it paid heavy price both in terms of speed of development of the language and popularity :-(.

Some of the main reasons that Python is so popular is so called "first language effect"-- language that university taught to students as the  first language has tremendous advantage over alternatives.

Python was adopted as the first language for teaching programming at many universities, and that produces steady stream of language users and some enthusiasts. 

Python interpreter has interactive mode, suitable for small examples and it more or less forgiving (no obligatory semicolon at the end of the statements like in Perl, not C-style curvy bracket delimited blocks (the source of a lot of grief for beginners as missing  curvy bracket is difficult to locate). Also it has more or less regular lexical structure and simple syntax (due to its very complex lexical structure and syntax Perl is as horrible as a beginner language; although semantic if Perl is better, more understandable,  for beginners then semantic of Python).  And after the language found its niche in intro university courses  the proverb "nothing succeed like success" is fully applicable.

Overcomplexity

Despite some claim that Python adhere to simplicity this is simply is not tire. This is a large non-orthogonal language, not that different  in this respect from Perl just with the different set of warts. Python is a large language and a decent textbook such as  Learning Python, 5th Edition by Mark Lutz (2013) is  over thousand pages. Modest intro like Introducing Python Modern Computing in Simple Packages by Bill Lubanovic is 484 pages. Slightly more expanded intro  Python Essential Reference (4th Edition)  by David Beazley is over 700 pagers. O'Reilly cookbook is 706 pages. And so on and so forth. Humans can't learn such a large language and need to operate with  a subset.

Both Perl and Python belong to the class of language with an attitude of "whatever gets the job done" although Python pretends (just pretends) to follow KISS principle: on the words (but  not in practice) Python designers seems preferring simplicity and consistency in design to flexibility that Perl advocates.  

2.7 vs. 3.6 problem

There is no single Python language. There are two dialects which are often called 3.x and 2.x.

Currently Python 2.7 is the dominant version of Python for scientific and engineering computing (although the standard version that comes with RHEL 6.x is still python 2.6). 64-bit version is dominant . But 3.x version is promoted by new books and gain in popularity too.  It is now taught in universities, which instantly pushed it into mainstream. Infatuation with Java in US universities ended, and ended for good, because there is nothing interesting in Java -- it is one step forward and two step back, kind of Cobol for XXI century.

Python 3 has  better support for coroutines (here is quote form Fluent Python, chapter 16):

The infrastructure for coroutines appeared in PEP 342 — Coroutines via Enhanced Generators, implemented in Python 2.5 (2006): since then, the yield keyword can be used in an expression, and the .send(value) method was added to the generator API. Using .send(…), the caller of the generator can post data that then becomes the value of the yield expression inside the generator function. This allows a generator to be used as a coroutine: a procedure that collaborates with the caller, yielding and receiving values from the caller.

In addition to .send(…), PEP 342 also added .throw(…) and .close() methods that respectively allow the caller to throw an exception to be handled inside the generator, and to terminate it. These features are covered in the next section and in “Coroutine Termination and Exception Handling”.

The latest evolutionary step for coroutines came with PEP 380 - Syntax for Delegating to a Subgenerator, implemented in Python 3.3 (2012). PEP 380 made two syntax changes to generator functions, to make them more useful as coroutines:

These latest changes will be addressed in “Returning a Value from a Coroutine” and “Using yield from”.

Let’s follow the established tradition of Fluent Python and start with some very basic facts and examples, then move into increasingly mind-bending features.
 

Modules and OO

The key Python feature is the ability to user modules. In this sense it is a derivative of Modular 3.  OO features are bolted on top of this.

While Python provides OO features, like C++ can be used without them.  They were added to language, not present from the very beginning. And like in many other languages with OO features they became fashionable and promoted the language. Also OO features are badly abused in Python scripts -- I saw many cases when Linux/Unix maintenance script were written using OO features. which makes them less maintainable and  the code more verbose. 

While pointers to memory structures (aka objects) is how OO is implemented, unlike Perl Python it does not provide pointers as a separate data type.  You can use pointers via object oriented framework but generally this is a perversion.  In a decent language pointers should be present as a separate data type.

Python is shipped with all version of Linux not other Unix flavors

Currently Python is shipped as standard component only with Linux and FreeBSD. Nether Solaris, not AIX or HP-UX include Python by default (but they do include Perl).

Quantity and quality of library of available modules

By the number and quality of available modules Python is now competitive and in some areas exceeds Perl with its famous CPAN. Like Perl, Python also has large library of standard modules shipped with the interpreter. Among them

But Python also faces competition from more modern languages such  as Ruby and R. Although still less popular Ruby competes with Python, especially for among programmers who value coroutines paradigm of software development (and it is really paradigm, not just a language feature). Python provides generators that are close enough, but still...

Ecosystem

Currently Python has the most developed ecosystem among all scripting languages with a dozen of high quality IDE available (Perl has almost none, although Komodo Edit can be used as a proxy) .

Starting from version 2.7 debugger is decent and supports almost the same set of operation as famous Perl debugger. In other words at last it manage to bridge the gap with Perl debugger.  Better late then never ;-).

There are probably more  books published about Python then any other scripting language. In 2017 books about Python are still baked like there is no tomorrow, but most of them are of poor or average quality.  Especially written  by OO zealots. This amount of "junk books" is an interesting and pretty unique feature of the language.

Many Python books should be avoided at all costs as they do not explain the language by obscure it.  You are warned.

One sign of the popularity of a scripting language is availability of editors which use it as macro language. Here Python outshine Perl by several orders of magnitude

see PythonEditors - Python Wiki

Komodo is a high quality middleweight free editor  that supports writing macros in Python.  See Macro API Reference

Python get a foot into numeric computing

Python also get a foot into numeric computing and is via SciPy/NumPy. It is now widely used in molecular modeling area, which was previously dominated by compiled language (with Fortran being the major). In genomic applications it also managed partially displace Perl (although quality of regular expression integration into the language is much lower in Python), but now R is pushing Python out in this domain.

Indentation as a proxy for nesting -- questionable design decision

Python imitates FORTRAN 4 -- it uses an indentation to denote nesting, similarly like Fortran 4 used it to distinguish between labels and statements ;-).

That creates problems  if tabs are used as editor and Python interpreter might have different settings for tabs. And that can screw nesting of statements.  This also created problems with diffs and putting patches with patch. 

Multiline statements in Python are iether detected by unbalanced brackets or can be explicitly marked with a backslash. Ability to close multiple blocks by just changing indentation is a plus only for short program, visible on  the screen. At the  same time possibility of mixing tabs and blanks for indentation is a horror show. You need specifically check if your Python program accidentally contain tabs and convert them to blanks. 

By relegating block brackets to the lexical level of blank space and comments Python failed to make a necessary adjustment and include pretty printer into interpreter (with the possibility of creating pretty printed program from pseudo-comments). Such a pretty printer actually needs to understand two things: the format of comments that suggest indenting like

#{ <label>

#} <label>

and the current number of spaces in the tab like  pragma tab = 4. The interesting possibility is that in pretty printed program those comments can be removed and after a reading pretty printed program into the editor reinstated automatically. Such feature can be implemented in any scriptable editor.

My impression is that few people understand that C  solution for blocks ({ } blocks) was pretty weak in comparison with its prototype language (PL/1): it does not permit nice labeled closer of blocks like

A:do ... end A;

in PL/1. IMHO introduction of a pretty printer as a standard feature of both a compiler and a GUI environment is long overdue and here Python can make its mark.

By aditing indentation as proxy for nesting Python actually encourages a programmer to use a decent editor, but we knew that already, right?  This design decision also  narrows down the possible range of coding styles and  automatically leads to more compact (as for the number of lexical tokens) programs (deletion of curly brackets usually help to lessen the number of lines in C or Perl program by 20% or more). 

Difficulties of adapting to the language for Perl programmers

Although Python as a  scripting language used Perl as prototype and its features are generally similar to Perl, Perl programmers experience difficulties adapting to the language.  They're not overly similar in implementation details, nor even remotely similar in syntax. Their extension mechanisms also took different directions. 

Final notes

We all understand that in real life better language seldom win (look at Java).  Luck plays tremendous role in  determining languages popularity. Best commercially supported language that satisfy current fashion has better chances. Python managed  to ride the wave of enthusiasm toward OO programming, which (by-and-large) unfair relegated Perl to the second class languages.  And it is not a bad scripting language so in a way success of Python is our success too.

Python now also has several different implementation of the interpreter, which are a clear sign of both popularity and maturity of the language:  Along with CPython interpreter (which is standard) there is quite popular Jython  which uses JVM and thus integrates well with Java, and Iron Python which is Microsoft implementation (Python -- programming language)

The mainstream Python implementation, also known as CPython, is written in C compliant to the C89 standard, and is distributed with a large standard library written in a mixture of C and Python. CPython ships for a large number of supported platforms, including Microsoft Windows and most modern Unix-like systems. CPython was intended from almost its very conception to be cross-platform; its use and development on esoteric platforms such as Amoeba alongside more conventional ones like Unix or Macintosh has greatly helped in this regard.

Stackless Python is a significant fork of CPython that implements microthreads. It can be expected to run on approximately the same platforms that CPython runs on.

There are two other major implementations: Jython for the Java platform, and IronPython for the .NET platform. PyPy is an experimental self-hosting implementation of Python, in Python, that can output a variety of types of bytecode, object code and intermediate languages.

Several programs exist to package Python programs into standalone executables, including py2exe, PyInstaller, cx_Freeze and py2app.

Many Python programs can run on different Python implementations, on such disparate operating systems and execution environments, without change. In the case of the implementations running on top of the Java virtual machine or the Common Language Runtime, the platform-independence of these systems is harnessed by their respective Python implementation.

Many third-party libraries for Python (and even some first-party ones) are only available on Windows, Linux, BSD, and Mac OS X.

There is also a dialect called Stackless Python which adds support for coroutines, communication channels and task serialization.

Python also has better interface with C programs than Perl, which allow to write extension modules in C.

Nikolai Bezroukov


Top Visited
Switchboard
Latest
Past week
Past month

NEWS CONTENTS

Old News ;-)

[Nov 16, 2017] Effective Python Item 40 Consider Coroutines to Run Many Functions Concurrently

Nov 16, 2017 | www.informit.com

Threads give Python programmers a way to run multiple functions seemingly at the same time (see Item 37: "Use Threads for Blocking I/O, Avoid for Parallelism"). But there are three big problems with threads:

Python can work around all these issues with coroutines . Coroutines let you have many seemingly simultaneous functions in your Python programs. They're implemented as an extension to generators. The cost of starting a generator coroutine is a function call. Once active, they each use less than 1 KB of memory until they're exhausted.

Coroutines work by enabling the code consuming a generator to send a value back into the generator function after each yield expression. The generator function receives the value passed to the send function as the result of the corresponding yield expression.

def my_coroutine():
    while True:
        received = yield
        print('Received:', received)

it = my_coroutine()
next(it)             # Prime the coroutine
it.send('First')
it.send('Second')

>>>
Received: First
Received: Second

The initial call to next is required to prepare the generator for receiving the first send by advancing it to the first yield expression. Together, yield and send provide generators with a standard way to vary their next yielded value in response to external input.

For example, say you want to implement a generator coroutine that yields the minimum value it's been sent so far. Here, the bare yield prepares the coroutine with the initial minimum value sent in from the outside. Then the generator repeatedly yields the new minimum in exchange for the next value to consider.

def minimize():
    current = yield
    while True:
        value = yield current
        current = min(value, current)

The code consuming the generator can run one step at a time and will output the minimum value seen after each input.

it = minimize()
next(it)            # Prime the generator
print(it.send(10))
print(it.send(4))
print(it.send(22))
print(it.send(-1))

>>>
10
4
4
-1

The generator function will seemingly run forever, making forward progress with each new call to send . Like threads, coroutines are independent functions that can consume inputs from their environment and produce resulting outputs. The difference is that coroutines pause at each yield expression in the generator function and resume after each call to send from the outside. This is the magical mechanism of coroutines.

This behavior allows the code consuming the generator to take action after each yield expression in the coroutine. The consuming code can use the generator's output values to call other functions and update data structures. Most importantly, it can advance other generator functions until their next yield expressions. By advancing many separate generators in lockstep, they will all seem to be running simultaneously, mimicking the concurrent behavior of Python threads.

The Game of Life

Let me demonstrate the simultaneous behavior of coroutines with an example. Say you want to use coroutines to implement Conway's Game of Life. The rules of the game are simple. You have a two-dimensional grid of an arbitrary size. Each cell in the grid can either be alive or empty.

ALIVE = '*'
EMPTY = '-'

The game progresses one tick of the clock at a time. At each tick, each cell counts how many of its neighboring eight cells are still alive. Based on its neighbor count, each cell decides if it will keep living, die, or regenerate. Here's an example of a 5×5 Game of Life grid after four generations with time going to the right. I'll explain the specific rules further below.

  0   |   1   |   2   |   3   |   4
----- | ----- | ----- | ----- | -----
-*--- | --*-- | --**- | --*-- | -----
--**- | --**- | -*--- | -*--- | -**--
---*- | --**- | --**- | --*-- | -----
----- | ----- | ----- | ----- | -----

I can model this game by representing each cell as a generator coroutine running in lockstep with all the others.

To implement this, first I need a way to retrieve the status of neighboring cells. I can do this with a coroutine named count_neighbors that works by yielding Query objects. The Query class I define myself. Its purpose is to provide the generator coroutine with a way to ask its surrounding environment for information.

Query = namedtuple('Query', ('y', 'x'))

The coroutine yields a Query for each neighbor. The result of each yield expression will be the value ALIVE or EMPTY . That's the interface contract I've defined between the coroutine and its consuming code. The count_neighbors generator sees the neighbors' states and returns the count of living neighbors.

def count_neighbors(y, x):
    n_ = yield Query(y + 1, x + 0)  # North
    ne = yield Query(y + 1, x + 1)  # Northeast
    # Define e_, se, s_, sw, w_, nw ...
    # ...
    neighbor_states = [n_, ne, e_, se, s_, sw, w_, nw]
    count = 0
    for state in neighbor_states:
        if state == ALIVE:
            count += 1
    return count

I can drive the count_neighbors coroutine with fake data to test it. Here, I show how Query objects will be yielded for each neighbor. count_neighbors expects to receive cell states corresponding to each Query through the coroutine's send method. The final count is returned in the StopIteration exception that is raised when the generator is exhausted by the return statement.

it = count_neighbors(10, 5)
q1 = next(it)                  # Get the first query
print('First yield: ', q1)
q2 = it.send(ALIVE)            # Send q1 state, get q2
print('Second yield:', q2)
q3 = it.send(ALIVE)            # Send q2 state, get q3
# ...
try:
    count = it.send(EMPTY)     # Send q8 state, retrieve count
except StopIteration as e:
    print('Count: ', e.value)  # Value from return statement
>>>
First yield:  Query(y=11, x=5)
Second yield: Query(y=11, x=6)
...
Count:  2

Now I need the ability to indicate that a cell will transition to a new state in response to the neighbor count that it found from count_neighbors . To do this, I define another coroutine called step_cell . This generator will indicate transitions in a cell's state by yielding Transition objects. This is another class that I define, just like the Query class.

Transition = namedtuple('Transition', ('y', 'x', 'state'))

The step_cell coroutine receives its coordinates in the grid as arguments. It yields a Query to get the initial state of those coordinates. It runs count_neighbors to inspect the cells around it. It runs the game logic to determine what state the cell should have for the next clock tick. Finally, it yields a Transition object to tell the environment the cell's next state.

def game_logic(state, neighbors):
    # ...

def step_cell(y, x):
    state = yield Query(y, x)
    neighbors = yield from count_neighbors(y, x)
    next_state = game_logic(state, neighbors)
    yield Transition(y, x, next_state)

Importantly, the call to count_neighbors uses the yield from expression. This expression allows Python to compose generator coroutines together, making it easy to reuse smaller pieces of functionality and build complex coroutines from simpler ones. When count_neighbors is exhausted, the final value it returns (with the return statement) will be passed to step_cell as the result of the yield from expression.

Now, I can finally define the simple game logic for Conway's Game of Life. There are only three rules.

def game_logic(state, neighbors):
    if state == ALIVE:
        if neighbors < 2:
            return EMPTY     # Die: Too few
        elif neighbors > 3:
            return EMPTY     # Die: Too many
    else:
        if neighbors == 3:
            return ALIVE     # Regenerate
    return state

I can drive the step_cell coroutine with fake data to test it.

it = step_cell(10, 5)
q0 = next(it)           # Initial location query
print('Me:      ', q0)
q1 = it.send(ALIVE)     # Send my status, get neighbor query
print('Q1:      ', q1)
# ...
t1 = it.send(EMPTY)     # Send for q8, get game decision
print('Outcome: ', t1)

>>>
Me:       Query(y=10, x=5)
Q1:       Query(y=11, x=5)
...
Outcome:  Transition(y=10, x=5, state='-')

The goal of the game is to run this logic for a whole grid of cells in lockstep. To do this, I can further compose the step_cell coroutine into a simulate coroutine. This coroutine progresses the grid of cells forward by yielding from step_cell many times. After progressing every coordinate, it yields a TICK object to indicate that the current generation of cells have all transitioned.

TICK = object()

def simulate(height, width):
    while True:
        for y in range(height):
            for x in range(width):
                yield from step_cell(y, x)
        yield TICK

What's impressive about simulate is that it's completely disconnected from the surrounding environment. I still haven't defined how the grid is represented in Python objects, how Query , Transition , and TICK values are handled on the outside, nor how the game gets its initial state. But the logic is clear. Each cell will transition by running step_cell . Then the game clock will tick. This will continue forever, as long as the simulate coroutine is advanced.

This is the beauty of coroutines. They help you focus on the logic of what you're trying to accomplish. They decouple your code's instructions for the environment from the implementation that carries out your wishes. This enables you to run coroutines seemingly in parallel. This also allows you to improve the implementation of following those instructions over time without changing the coroutines.

Now, I want to run simulate in a real environment. To do that, I need to represent the state of each cell in the grid. Here, I define a class to contain the grid:

class Grid(object):
    def __init__(self, height, width):
        self.height = height
        self.width = width
        self.rows = []
        for _ in range(self.height):
            self.rows.append([EMPTY] * self.width)

    def __str__(self):
        # ...

The grid allows you to get and set the value of any coordinate. Coordinates that are out of bounds will wrap around, making the grid act like infinite looping space.

    def query(self, y, x):
        return self.rows[y % self.height][x % self.width]

    def assign(self, y, x, state):
        self.rows[y % self.height][x % self.width] = state

At last, I can define the function that interprets the values yielded from simulate and all of its interior coroutines. This function turns the instructions from the coroutines into interactions with the surrounding environment. It progresses the whole grid of cells forward a single step and then returns a new grid containing the next state.

def live_a_generation(grid, sim):
    progeny = Grid(grid.height, grid.width)
    item = next(sim)
    while item is not TICK:
        if isinstance(item, Query):
            state = grid.query(item.y, item.x)
            item = sim.send(state)
        else:  # Must be a Transition
            progeny.assign(item.y, item.x, item.state)
            item = next(sim)
    return progeny

To see this function in action, I need to create a grid and set its initial state. Here, I make a classic shape called a glider.

grid = Grid(5, 9)
grid.assign(0, 3, ALIVE)
# ...
print(grid)

>>>
---*-----
----*----
--***----
---------
---------

Now I can progress this grid forward one generation at a time. You can see how the glider moves down and to the right on the grid based on the simple rules from the game_logic function.

class ColumnPrinter(object):
    # ...

columns = ColumnPrinter()
sim = simulate(grid.height, grid.width)
for i in range(5):
    columns.append(str(grid))
    grid = live_a_generation(grid, sim)

print(columns)

>>>
    0     |     1     |     2     |     3     |     4
---*----- | --------- | --------- | --------- | ---------
----*---- | --*-*---- | ----*---- | ---*----- | ----*----
--***---- | ---**---- | --*-*---- | ----**--- | -----*---
--------- | ---*----- | ---**---- | ---**---- | ---***---
--------- | --------- | --------- | --------- | ---------

The best part about this approach is that I can change the game_logic function without having to update the code that surrounds it. I can change the rules or add larger spheres of influence with the existing mechanics of Query , Transition , and TICK . This demonstrates how coroutines enable the separation of concerns, which is an important design principle.

Coroutines in Python 2

Unfortunately, Python 2 is missing some of the syntactical sugar that makes coroutines so elegant in Python 3. There are two limitations. First, there is no yield from expression. That means that when you want to compose generator coroutines in Python 2, you need to include an additional loop at the delegation point.

# Python 2
def delegated():
    yield 1
    yield 2

def composed():
    yield 'A'
    for value in delegated():  # yield from in Python 3
        yield value
    yield 'B'

print list(composed())

>>>
['A', 1, 2, 'B']

The second limitation is that there is no support for the return statement in Python 2 generators. To get the same behavior that interacts correctly with try / except / finally blocks, you need to define your own exception type and raise it when you want to return a value.

# Python 2
class MyReturn(Exception):
    def __init__(self, value):
        self.value = value

def delegated():
    yield 1
    raise MyReturn(2)  # return 2 in Python 3
    yield 'Not reached'

def composed():
    try:
        for value in delegated():
            yield value
    except MyReturn as e:
        output = e.value
    yield output * 4

print list(composed())

>>>
[1, 8]
Things to Remember

[Nov 16, 2017] Python generators and coroutines - Stack Overflow

Notable quotes:
"... Edit: I recommend using Greenlet . But if you're interested in a pure Python approach, read on. ..."
"... at a language level ..."
"... To anyone reading this in 2015 or later, the new syntax is 'yield from' ( PEP 380 ) and it allows true coroutines in Python >3.3 ..."
Nov 16, 2017 | stackoverflow.com

Python generators and coroutines Ask Question up vote down vote favorite 6

Giuseppe Maggiore ,May 10, 2011 at 10:25

I am studying coroutines and generators in various programming languages.

I was wondering if there is a cleaner way to combine together two coroutines implemented via generators than yielding back at the caller whatever the callee yields?

Let's say that we are using the following convention: all yields apart from the last one return null, while the last one returns the result of the coroutine. So, for example, we could have a coroutine that invokes another:

def A():
  # yield until a certain condition is met
  yield result

def B():
  # do something that may or may not yield
  x = bind(A())
  # ...
  return result

in this case I wish that through bind (which may or may not be implementable, that's the question) the coroutine B yields whenever A yields until A returns its final result, which is then assigned to x allowing B to continue.

I suspect that the actual code should explicitly iterate A so:

def B():
  # do something that may or may not yield
  for x in A(): ()
  # ...
  return result

which is a tad ugly and error prone...

PS: it's for a game where the users of the language will be the designers who write scripts (script = coroutine). Each character has an associated script, and there are many sub-scripts which are invoked by the main script; consider that, for example, run_ship invokes many times reach_closest_enemy, fight_with_closest_enemy, flee_to_allies, and so on. All these sub-scripts need to be invoked the way you describe above; for a developer this is not a problem, but for a designer the less code they have to write the better!

S.Lott ,May 10, 2011 at 10:38

This is all covered on the Python web site. python.org/dev/peps/pep-0342 , python.org/dev/peps/pep-0334 and numerous blogs cover this. eecho.info/Echo/python/coroutines-python . Please Google, read, and then ask specific questions based on what you've read. – S.Lott May 10 '11 at 10:38

S.Lott ,May 10, 2011 at 13:04

I thought the examples clearly demonstrated idiomatic. Since I'm unable to understand what's wrong with the examples, could you state which examples you found to be unclear? Which examples were confusing? Can you be more specific on how all those examples where not able to show idiomatic Python? – S.Lott May 10 '11 at 13:04

Giuseppe Maggiore ,May 10, 2011 at 13:09

I've read precisely those articles, and the PEP-342 leaves me somewhat confused: is it some actual extension that is currently working in Python? Is the Trampoline class shown there part of the standard libraries of the language? BTW, my question was very precise, and it was about the IDIOMATIC way to pass control around coroutines. The fact that I can read about a ton of ways to do so really does not help. Neither does your snarkiness... – Giuseppe Maggiore May 10 '11 at 13:09

Giuseppe Maggiore ,May 10, 2011 at 13:11

Idiomatic is about the "standard" way to perform some function; there is absolutely nothing wrong with iterating the results of a nested coroutine, but there are examples in the literature of programming languages where yielding automatically climbs down the call stack and so you do not need to re-yield at each caller, hence my curiosity if this pattern is covered by sintactic sugar in Python or not! – Giuseppe Maggiore May 10 '11 at 13:11

S.Lott ,May 10, 2011 at 13:19

@Giuseppe Maggiore: "programming languages where yielding automatically climbs down the call stack" That doesn't sound like the same question. Are you asking for idiomatic Python -- as shown by numerous examples -- or are you asking for some other feature that's not shown in the Python examples but is shown in other languages? I'm afraid that I can't understand your question at all. Can you please clarify what you're really looking for? – S.Lott May 10 '11 at 13:19

blubb ,May 10, 2011 at 10:37

Are you looking for something like this?
def B():
   for x in A():
     if x is None:
       yield
     else:
       break

   # continue, x contains value A yielded

Giuseppe Maggiore ,May 10, 2011 at 12:59

Yes, that is what I am doing. My question is if this is the idiomatic way or if there is some syntactic construct that is capable of hiding this pattern which recurs very often in my application. – Giuseppe Maggiore May 10 '11 at 12:59

blubb ,May 10, 2011 at 13:31

@Guiseppe Maggiore: I'm not aware of any such constructs. However, it seems strange that you need this pattern often... I can't think of many valid used cases off the top of my head. If you give more context information, maybe we can propose an alternative solution which is more elegant over all? – blubb May 10 '11 at 13:31

Giuseppe Maggiore ,May 10, 2011 at 15:17

It's for a game where the users of the language will be the designers who write scripts (script = coroutine). Each character has an associated script, and there are many sub-scripts which are invoked by the main script; consider that, for example, run_ship invokes many times reach_closest_enemy, fight_with_closest_enemy, flee_to_allies, and so on. All these sub-scripts need to be invoked the way you describe above; for a developer this is not a problem, but for a designer the less code they have to write the better! – Giuseppe Maggiore May 10 '11 at 15:17

blubb ,May 10, 2011 at 15:57

@Guiseppe Maggiore: I'd propose you add that last comment to the question so that other get a chance of answering it, too... – blubb May 10 '11 at 15:57

Simon Radford ,Nov 11, 2011 at 0:24

Edit: I recommend using Greenlet . But if you're interested in a pure Python approach, read on.

This is addressed in PEP 342 , but it's somewhat tough to understand at first. I'll try to explain simply how it works.

First, let me sum up what I think is the problem you're really trying to solve.

Problem

You have a callstack of generator functions calling other generator functions. What you really want is to be able to yield from the generator at the top, and have the yield propagate all the way down the stack.

The problem is that Python does not ( at a language level ) support real coroutines, only generators. (But, they can be implemented.) Real coroutines allow you to halt an entire stack of function calls and switch to a different stack. Generators only allow you to halt a single function. If a generator f() wants to yield, the yield statement has to be in f(), not in another function that f() calls.

The solution that I think you're using now, is to do something like in Simon Stelling's answer (i.e. have f() call g() by yielding all of g()'s results). This is very verbose and ugly, and you're looking for syntax sugar to wrap up that pattern. Note that this essentially unwinds the stack every time you yield, and then winds it back up again afterwards.

Solution

There is a better way to solve this problem. You basically implement coroutines by running your generators on top of a "trampoline" system.

To make this work, you need to follow a couple patterns: 1. When you want to call another coroutine, yield it. 2. Instead of returning a value, yield it.

so

def f():
    result = g()
    #  
    return return_value

becomes

def f():
    result = yield g()
    #  
    yield return_value

Say you're in f(). The trampoline system called f(). When you yield a generator (say g()), the trampoline system calls g() on your behalf. Then when g() has finished yielding values, the trampoline system restarts f(). This means that you're not actually using the Python stack; the trampoline system manages a callstack instead.

When you yield something other than a generator, the trampoline system treats it as a return value. It passes that value back to the caller generator through the yield statement (using .send() method of generators).

Comments

This kind of system is extremely important and useful in asynchronous applications, like those using Tornado or Twisted. You can halt an entire callstack when it's blocked, go do something else, and then come back and continue execution of the first callstack where it left off.

The drawback of the above solution is that it requires you to write essentially all your functions as generators. It may be better to use an implementation of true coroutines for Python - see below.

Alternatives

There are several implementations of coroutines for Python, see: http://en.wikipedia.org/wiki/Coroutine#Implementations_for_Python

Greenlet is an excellent choice. It is a Python module that modifies the CPython interpreter to allow true coroutines by swapping out the callstack.

Python 3.3 should provide syntax for delegating to a subgenerator, see PEP 380 .

gaborous ,Nov 9, 2012 at 10:04

Very useful and clear answer, thank's! However, when you say that standard Python coroutines essentially require to write all functions as generators, did you mean only first level functions or really all functions? As you said above, when yielding something other than a generator, the trampoline system still works, so theoretically we can just yield at the first-layer functions any other functions that may or may not be generators themselves. Am I right? – gaborous Nov 9 '12 at 10:04

Simon Radford ,Nov 21, 2012 at 21:37

All "functions" between the trampoline system and a yield must be written as generators. You can call regular functions normally, but then you can't effectively "yield" from that function or any functions it calls. Does that make sense / answer your question? – Simon Radford Nov 21 '12 at 21:37

Simon Radford ,Nov 21, 2012 at 21:39

I highly recommend using Greenlet - it's a true implementation of coroutines for Python, and you don't have to use any of these patterns I've described. The trampoline stuff is for people who are interested in how you can do it in pure Python. – Simon Radford Nov 21 '12 at 21:39

Nick Sweeting ,Jun 7, 2015 at 22:12

To anyone reading this in 2015 or later, the new syntax is 'yield from' ( PEP 380 ) and it allows true coroutines in Python >3.3 . – Nick Sweeting Jun 7 '15 at 22:12

[Nov 14, 2017] Masterminds of Programming Conversations with the Creators of Major Programming Languages

Notable quotes:
"... What differences are there between developing a programming language and developing a "common" software project? ..."
"... How do you debug a language? ..."
"... How do you decide when a feature should go in a library as an extension or when it needs to have support from the core language? ..."
"... I suppose there are probably features that you've looked at that you couldn't implement in Python other than by changing the language, but you probably rejected them. What criteria do you use to say this is something that's Pythonic, this is something that's not Pythonic? ..."
"... You have the "Zen of Python," but beyond that? ..."
"... Sounds almost like it's a matter of taste as much as anything ..."
"... There's an argument to make for parsimony there, but very much in the context of personal taste ..."
"... How did the Python Enhancement Proposal (PEP) process come about? ..."
"... Do you find that adding a little bit of formalism really helps crystallize the design decisions around Python enhancements? ..."
"... Do they lead to a consensus where someone can ask you to weigh in on a single particular crystallized set of expectations and proposals? ..."
"... What creates the need for a new major version? ..."
"... How did you choose to handle numbers as arbitrary precision integers (with all the cool advantages you get) instead of the old (and super common) approach to pass it to the hardware? ..."
"... Why do you call it a radical step? ..."
"... How did you adopt the "there should be one -- and preferably only one -- obvious way to do it" philosophy? ..."
"... What is your take on static versus dynamic typing? ..."
"... Are we moving toward hybrid typing? ..."
"... Why did you choose to support multiple paradigms? ..."
"... When you created the language, did you consider the type of programmers it might have attracted? ..."
"... How do you balance the different needs of a language that should be easy to learn for novices versus a language that should be powerful enough for experienced programmers to do useful things? Is that a false dichotomy? ..."
Nov 14, 2017 | www.amazon.com

The Pythonic Way

What differences are there between developing a programming language and developing a "common" software project?

Guido van Rossum : More than with most software projects, your most important users are programmers themselves. This gives a language project a high level of "meta" content. In the dependency tree of software projects, programming

How do you debug a language?

Guido : You don't. Language design is one area where agile development methodologies just don't make sense -- until the language is stable, few people want to use it, and you won't find the bugs in the language definition until you have so many users that it's too late to change things.

Of course there's plenty in the implementation that can be debugged like any old program, but the language design itself pretty much requires careful design up front, because the cost of bugs is so exorbitant.

How do you decide when a feature should go in a library as an extension or when it needs to have support from the core language?

Guido : Historically, I've had a pretty good answer for that. One thing I noticed very early on was that everybody wants their favorite feature added to the language, and most people are relatively inexperienced about language design. Everybody is always proposing "let's add this to the language," "let's have a statement that does X." In many cases, the answer is, "Well, you can already do X or something almost like X by writing these two or three lines of code, and it's not all that difficult." You can use a dictionary, or you can combine a list and a tuple and a regular expression, or write a little metaclass -- all of those things. I may even have had the original version of this answer from Linus, who seems to have a similar philosophy.

Telling people you can already do that and here is how is a first line of defense. The second thing is, "Well, that's a useful thing and we can probably write or you can probably write your own module or class, and encapsulate that particular bit of abstraction." Then the next line of defense is, "OK, this looks so interesting and useful that we'll actually accept it as a new addition to the standard library, and it's going to be pure Python." And then, finally, there are things that just aren't easy to do in pure Python and we'll suggest or recommend how to turn them into a C extension. The C extensions are the last line of defense before we have to admit, "Well, yeah, this is so useful and you really cannot do this, so we'll have to change the language."

There are other criteria that determine whether it makes more sense to add something to the language or it makes more sense to add something to the library, because if it has to do with the semantics of namespaces or that kind of stuff, there's really nothing you can do besides changing the language. On the other hand, the extension mechanism was made powerful enough that there is an amazing amount of stuff you can do from C code that extends the library and possibly even adds new built-in functionality without actually changing the language. The parser doesn't change. The parse tree doesn't change. The documentation for the language doesn't change. All your tools still work, and yet you have added new functionality to your system.

I suppose there are probably features that you've looked at that you couldn't implement in Python other than by changing the language, but you probably rejected them. What criteria do you use to say this is something that's Pythonic, this is something that's not Pythonic?

Guido : That's much harder. That is probably, in many cases, more a matter of a gut feeling than anything else. People use the word Pythonic and "that is Pythonic" a lot, but nobody can give you a watertight definition of what it means for something to be Pythonic or un-Pythonic.

You have the "Zen of Python," but beyond that?

Guido : That requires a lot of interpretation, like every good holy book. When I see a good or a bad proposal, I can tell if it is a good or bad proposal, but it's really hard to write a set of rules that will help someone else to distinguish good language change proposals from bad change proposals.

Sounds almost like it's a matter of taste as much as anything

Guido : Well, the first thing is always try to say "no," and see if they go away or find a way to get their itch scratched without changing the language. It's remarkable how often that works. That's more of a operational definition of "it's not necessary to change the language."

If you keep the language constant, people will still find a way to do what they need to do. Beyond that it's often a matter of use cases coming from different areas where there is nothing application-specific. If something was really cool for the Web, that would not make it a good feature to add to the language. If something was really good for writing shorter functions or writing classes that are more maintainable, that might be a good thing to add to the language. It really needs to transcend application domains in general, and make things simpler or more elegant.

When you change the language, you affect everyone. There's no feature that you can hide so well that most people don't need to know about. Sooner or later, people will encounter code written by someone else that uses it, or they'll encounter some obscure corner case where they have to learn about it because things don't work the way they expected.

Often elegance is also in the eye of the beholder. We had a recent discussion on one of the Python lists where people were arguing forcefully that using dollar instead of self-dot was much more elegant. I think their definition of elegance was number of keystrokes.

There's an argument to make for parsimony there, but very much in the context of personal taste

Guido : Elegance and simplicity and generality all are things that, to a large extent, depend on personal taste, because what seems to cover a larger area for me may not cover enough for someone else, and vice versa.

How did the Python Enhancement Proposal (PEP) process come about?

Guido : That's a very interesting historical tidbit. I think it was mostly started and championed by Barry Warsaw, one of the core developers. He and I started working together in '95, and I think around 2000, he came up with the suggestion that we needed more of a formal process around language changes.

I tend to be slow in these things. I mean I wasn't the person who discovered that we really needed a mailing list. I wasn't the person who discovered that the mailing list got unwieldy and we needed a newsgroup. I wasn't the person to propose that we needed a website. I was also not the person to propose that we needed a process for discussing and inventing language changes, and making sure to avoid the occasional mistake where things had been proposed and quickly accepted without thinking through all of the consequences.

At the time between 1995 and 2000, Barry, myself, and a few other core developers, Fred Drake, Ken Manheimer for a while, were all at CNRI, and one of the things that CNRI did was organize the IETF meetings. CNRI had this little branch that eventually split off that was a conference organizing bureau, and their only customer was the IETF. They later also did the Python conferences for a while, actually. Because of that it was a pretty easy boondoggle to attend IETF meetings even if they weren't local. I certainly got a taste of the IETF process with its RFCs and its meeting groups and stages, and Barry also got a taste of that. When he proposed to do something similar for Python, that was an easy argument to make. We consciously decided that we wouldn't make it quite as heavy-handed as the IETF RFCs had become by then, because Internet standards, at least some of them, affect way more industries and people and software than a Python change, but we definitely modeled it after that. Barry is a genius at coming up with good names, so I am pretty sure that PEP was his idea.

We were one of the first open source projects at the time to have something like this, and it's been relatively widely copied. The Tcl/Tk community basically changed the title and used exactly the same defining document and process, and other projects have done similar things.

Do you find that adding a little bit of formalism really helps crystallize the design decisions around Python enhancements?

Guido : I think it became necessary as the community grew and I wasn't necessarily able to judge every proposal on its value by itself. It has really been helpful for me to let other people argue over various details, and then come with relatively clear-cut conclusions.

Do they lead to a consensus where someone can ask you to weigh in on a single particular crystallized set of expectations and proposals?

Guido : Yes. It often works in a way where I initially give a PEP a thumb's up in the sense that I say, "It looks like we have a problem here. Let's see if someone figures out what the right solution is." Often they come out with a bunch of clear conclusions on how the problem should be solved and also a bunch of open issues. Sometimes my gut feelings can help close the open issues. I'm very active in the PEP process when it's an area that I'm excited about -- if we had to add a

What creates the need for a new major version?

Guido : It depends on your definition of major. In Python, we generally consider releases like 2.4, 2.5, and 2.6 "major" events, which only happen every 18–24 months. These are the only occasions where we can introduce new features. Long ago, releases were done at the whim of the developers (me, in particular). Early this decade, however, the users requested some predictability -- they objected against features being added or changed in "minor" revisions (e.g., 1.5.2 added major features compared to 1.5.1), and they wished the major releases to be supported for a certain minimum amount of time (18 months). So now we have more or less time-based major releases: we plan the series of dates leading up to a major release (e.g., when alpha and beta versions and release candidates are issued) long in advance, based on things like release manager availability, and we urge the developers to get their changes in well in advance of the final release date.

Features selected for addition to releases are generally agreed upon by the core developers, after (sometimes long) discussions on the merits of the feature and its precise specification. This is the PEP process: Python Enhancement Proposal, a document-base process not unlike the IETF's RFC process or the Java world's JSR process, except that we aren't quite as formal, as we have a much smaller community of developers. In case of prolonged disagreement (either on the merits of a feature or on specific details), I may end up breaking a tie; my tie-breaking algorithm is mostly intuitive, since by the time it is invoked, rational argument has long gone out of the window.

The most contentious discussions are typically about user-visible language features; library additions are usually easy (as they don't harm users who don't care), and internal improvements are not really considered features, although they are constrained by pretty stringent backward compatibility at the C API level.

Since the developers are typically the most vocal users, I can't really tell whether

There's also the concept of a radically major or breakthrough version, like 3.0. Historically, 1.0 was evolutionarily close to 0.9, and 2.0 was also a relatively small step from 1.6. From now on, with the much larger user base, such versions are rare indeed, and provide the only occasion for being truly incompatible with previous versions. Major versions are made backward compatible with previous major versions with a specific mechanism available for deprecating features slated for removal.

How did you choose to handle numbers as arbitrary precision integers (with all the cool advantages you get) instead of the old (and super common) approach to pass it to the hardware?

Guido : I originally inherited this idea from Python's predecessor, ABC. ABC used arbitrary precision rationals, but I didn't like the rationals that much, so I switched to integers; for reals, Python uses the standard floating-point representation supported by the hardware (and so did ABC, with some prodding).

Originally Python had two types of integers: the customary 32-bit variety ("int") and a separate arbitrary precision variety ("long"). Many languages do this, but the arbitrary precision variety is relegated to a library, like Bignum in Java and Perl, or GNU MP for C.

Previously, this would raise an OverflowError exception. There was once a time where the result would silently be truncated, but I changed it to raising an exception before ever letting others use the language. In early 1990, I wasted an afternoon debugging a short demo program I'd written implementing an algorithm that made non-obvious use of very large integers. Such debugging sessions are seminal experiences.

However, there were still certain cases where the two number types behaved slightly different; for example, printing an int in hexadecimal or octal format would produce an unsigned outcome (e.g., –1 would be printed as FFFFFFFF), while doing the same on the mathematically equal long would produce a signed outcome (–1, in this case). In Python 3.0, we're taking the radical step of supporting only a single integer type; we're calling it int , but the implementation is largely that of the old long type.

Why do you call it a radical step?

Guido : Mostly because it's a big deviation from current practice in Python. There was a lot of discussion about this, and people proposed various alternatives where two (or more) representations would be used internally, but completely or mostly hidden from end users (but not from C extension writers). That might perform a bit better, but in the end it was already a massive amount of work, and having two representations internally would just increase the effort of getting it right, and make interfacing to it from C

How did you adopt the "there should be one -- and preferably only one -- obvious way to do it" philosophy?

Guido : This was probably subconscious at first. When Tim definition of having one way (or one true way) to express something. For example, the XYZ coordinates of any point in 3D space are uniquely determined, once you've picked an origin and three basis vectors.

I also like to think that I'm doing most users a favor by not requiring them to choose between similar alternatives. You can contrast this with Java, where if you need a listlike data structure, the standard library offers many versions (a linked list, or an array list, and others), or C, where you have to decide how to implement your own list data type.

What is your take on static versus dynamic typing?

Guido : I wish I could say something simple like "

In some situations the verbosity of Java is considered a plus; it has enabled the creation of powerful code-browsing tools that can answer questions like "where is this variable changed?" or "who calls this method?" Dynamic languages make answering such questions harder, because it's often hard to find out the type of a method argument without analyzing every path through the entire codebase. I'm not sure how functional languages like Haskell support such tools; it could well be that you'd have to use essentially the same technique as for dynamic languages, since that's what type inferencing does anyway -- in my limited understanding!

Are we moving toward hybrid typing?

Guido : I expect there's a lot to say for some kind of hybrid. I've noticed that most large systems written in a statically typed language actually contain a significant subset that is essentially dynamically typed. For example, GUI widget sets and database APIs for Java often feel like they are fighting the static typing every step of the way, moving most correctness checks to runtime.

A hybrid language with functional and dynamic aspects might be quite interesting. I should add that despite Python's support for some functional tools like map() and lambda , Python does not have a functional-language subset: there is no type inferencing, and no opportunity for parallelization.

Why did you choose to support multiple paradigms?

Guido : I didn't really; Python supports procedural programming, to some extent, and OO. These two aren't so different, and Python's procedural style is still strongly influenced by objects (since the fundamental data types are all objects). Python supports a tiny bit of functional programming -- but it doesn't resemble any real functional language, and it never will. Functional languages are all about doing as much as possible at compile time -- the "functional" aspect means that the compiler can optimize things under a very strong guarantee that there are no side effects, unless explicitly declared. Python is about having the simplest, dumbest compiler imaginable, and the official runtime semantics actively discourage cleverness in the compiler like parallelizing loops or turning recursion into loops.

Python probably has the reputation of supporting functional programming based on the inclusion of lambda , map , filter , and reduce in the language, but in my eyes these are just syntactic sugar, and not the fundamental building blocks that they are in functional languages. The more fundamental property that Python shares with Lisp (not a functional language either!) is that functions are first-class objects, and can be passed around like any other object. This, combined with nested scopes and a generally Lisp-like approach to function state, makes it possible to easily implement concepts that superficially resemble concepts from functional languages, like currying, map, and reduce. The primitive operations that are necessary to implement those concepts are built are the primitive operations. You can write reduce() in a few lines of Python. Not so in a functional language.

When you created the language, did you consider the type of programmers it might have attracted?

Guido : Yes, but I probably didn't have enough imagination. I was thinking of professional programmers in a Unix or Unix-like environment. Early versions of the Python tutorial used a slogan something like "Python bridges the gap between C and shell programming," because that was where I was myself, and the people immediately around me. It never occurred to me that Python would be a

The fact that it was useful for teaching first principles of

How do you balance the different needs of a language that should be easy to learn for novices versus a language that should be powerful enough for experienced programmers to do useful things? Is that a false dichotomy?

Guido : Balance is the word. There are some well-known traps to avoid, like stuff that is thought to help novices but annoys

[Nov 09, 2017] Conversion of Perl to Python

Nov 09, 2017 | stackoverflow.com

I think you should rewrite your code. The quality of the results of a parsing effort depends on your Perl coding style. I think the quote below sums up the theoretical side very well. From Wikipedia: Perl in Wikipedia

Perl has a Turing-complete grammar because parsing can be affected by run-time code executed during the compile phase.[25] Therefore, Perl cannot be parsed by a straight Lex/Yacc lexer/parser combination. Instead, the interpreter implements its own lexer, which coordinates with a modified GNU bison parser to resolve ambiguities in the language.

It is often said that "Only perl can parse Perl," meaning that only the Perl interpreter (perl) can parse the Perl language (Perl), but even this is not, in general, true. Because the Perl interpreter can simulate a Turing machine during its compile phase, it would need to decide the Halting Problem in order to complete parsing in every case. It's a long-standing result that the Halting Problem is undecidable, and therefore not even Perl can always parse Perl. Perl makes the unusual choice of giving the user access to its full programming power in its own compile phase. The cost in terms of theoretical purity is high, but practical inconvenience seems to be rare.

Other programs that undertake to parse Perl, such as source-code analyzers and auto-indenters, have to contend not only with ambiguous syntactic constructs but also with the undecidability of Perl parsing in the general case. Adam Kennedy's PPI project focused on parsing Perl code as a document (retaining its integrity as a document), instead of parsing Perl as executable code (which not even Perl itself can always do). It was Kennedy who first conjectured that, "parsing Perl suffers from the 'Halting Problem'."[26], and this was later proved.[27]

Starting in 5.10, you can compile perl with the experimental Misc Attribute Decoration enabled and set the PERL_XMLDUMP environment variable to a filename to get an XML dump of the parse tree (including comments - very helpful for language translators). Though as the doc says, this is a work in progress.

Looking at the PLEAC stuff, what we have here is a case of a rote translation of a technique from one language causing another to look bad. For example, its rare in Perl to work character-by-character. Why? For one, its a pain in the ass. A fair cop. For another, you can usually do it faster and easier with a regex. One can reverse the OP's statement and say "in Perl, regexes are so easy that most of the time other string manipulation is not needed". Anyhow, the OP's sentiment is correct. You do things differently in Perl than in Python so a rote translator would produce nasty code. – Schwern Apr 8 '10 at 11:47

down vote Converting would require writing a Perl parser, semantic checker, and Python code generator.

Not practical. Perl parsers are hard enough for the Perl teams to get right. You'd be better off translating Perl to Python from the Perl AST (opcodes) using the Perl Opcode or related modules.

http://perldoc.perl.org/Opcode.html

Some notations do not map from Perl to Python without some work. Perl's closures are different, for example. So is its regex support.

In short, either convert it by hand, or use some integration modules to call Python from Perl or vice-versa

[Nov 07, 2017] Is PyCharm good - Quora

Nov 07, 2017 | www.quora.com

Cody Jackson , Python book author ( https://python-ebook.blogspot.com ) Answered Sep 11

I stumbled upon PyCharm a few years ago when my editor of choice (Stani's Python Editor) was no longer maintained. I haven't looked back.

I used the community edition for many years then decided to purchase a copy. While I don't necessarily need all the functionality of the paid version, I want to support the company in their work.

The PEP 8 notifications are nice to have. While PEP 8 is more of a guideline, it certainly helps ensure code looks nice and is easy to work with.

What's better, IMO, is the ability to load anything you want without having to explicitly download it. Import a module that isn't already on your system? PyCharm will let you know and offer to download it for you. Very handy.

I used to use GitKraken for GitHub work but the built-in VCS tools in PyCharm are just as easy to use, so I haven't bothered to download GitKraken for several months now. PyCharm highlights your modified files using color codes, so you know what you have updated, what's new, etc. so you know exactly what is going to be added in your next push. It also shows you what has changed between the different files using diff, which is handy.

PyCharm has built-in support for many different frameworks, the paid version obviously having more support. However, the free version includes Django, HTML, CSS, and JavaScript, which is sufficient for most people.

While the paid version has changed from a perpetual licenses to a subscription model, the monthly cost is only $8 per month for an individual, with certain discounts available.

Overall, PyCharm is the best proprietary Python editor and, unless you prefer completely FOSS software, there is no reason not to use it.

Yosef Dishinger , I dream in Python Answered Sep 28

The other answers have already said most of it, but I would just add that the search and code discovery features of PyCharm are superior to anything else I've used.

I work on a pretty large codebase, and with PyCharm you can search throughout the entire project, or even multiple projects, for a given string. Now it's true that other editors also have this feature, but PyCharm adds something here that other editors don't.

It lets you edit the code where the reference was found, in a panel within the search results window, and simply go through each search result one by one and see and modify the code in each section as you go, without needing to open the different files on their own.

At times when I've needed to do major refactoring this has been a lifesaver. It increased my productivity dramatically.

There are a lot of really nice editors out there, but I haven't come across anything like PyCharm for taming large codebases.

Edward Moseley , Python for programming, R for stats, C/C++ for microcontrollers Answered Aug 27 2016

I'm very much in agreement with User-9321510923064044481

If you begin to use a library that you don't have installed, PyCharm will let you know and makes the installation process really seamless. RStudio could actually probably take a page out of PyCharm's playbook, there.

I use the integrated python console very frequently for prototyping.

There's also this "Tip of the day" popup that I always mean to shut off but well sometimes they are good tips.

This may be nit-picky, but I especially agree that I don't use the integrated VCS , and until they find a more elegant way to integrate it I will stick to git on my command line.

[Nov 07, 2017] How to use Python interactively in PyCharm

Nov 07, 2017 | www.quora.com

Tony Flury , Freelance s/w developer Answered Apr 2

PyCharn when it starts will also start a python terminal as part of the project window. Look along the bottom where you will have tabs such as console and terminal.

PyCharn also offers integration with Jupiter notebook, but I haven't tried to use that feature yet.

Zdenko Hrcek , enjoying programming in Python Answered Apr 2

In main menu under Tools there is "Python console" option

Related Questions More Answers Below

[Nov 07, 2017] Should I use PyCharm for programming Python

Nov 07, 2017 | www.quora.com

AP Rajshekhar , Knows Java, Python, Ruby, Go; dabbled in Qt and GTK# Answered Sep 24, 2016

As with any other language, one does not need an IDE, which PyCharm is. However, it has been my experience that having an IDE improves productivity. Same is true with PyCharm.

If you are developing small applications that does not need git integration or PEP8 standards conformation, then you don't need PyCharm However, if you need any of the above, and do not want to use multiple tools (flake8, git-cli/git-cola) manually, then PyCharm is a good choice as it provides the following, apart from autocomplete, from within the IDE:

So, Pycharm improves your productivity quite a bit. Dominic Christoph , Met cofounders at a local meetup Updated Apr 5

It's obviously not necessary, and there are other free editors and IDEs. But in my experience, it is the best option.

I've used both Vim and Emacs and played with Sublime and Atom a bit. Those four editors allow you to highly customize your programming environment. Which some feel is a necessity.

They're all great, but you will miss out on some features that no one (that I know of; if you do, please share) has been able to properly recreate in a regular editor. Mainly, intelligent code navigation and completion. These are the most useful features that I've used, and PyCharm does them **almost** perfectly.

You'll spend much more time navigating code than you will typing code, so it's very helpful to be able to hit a keyboard shortcut and jump to a variable or method's definition/declaration. When you are typing, the intelligent autocomplete will be a big help as well. It's much more useable than the completion engines in editors because it only provides completions which are in scope. There're also Ctags and Gtags available for text editors but they are harder to use, must be customized for every language, and with any medium to large sized project work poorly. Though YMMV.

When it comes down to it, I prefer having features that work really well than the ability to customize. Download the community edition and see for yourself if it works for you. Especially for a beginner, it will save you the time of learning tools, which isn't as important as learning the language, because the UI is self-explanatory.

P.S.

I would find it unusable without the IdeaVim plugin. The keybindings of Vim are just too good to give up.

I should also mention that Jetbrains IDEs are very customizable themselves. The IdeaVim plugin even has a dotfile.

You'll also find videos on YouTube where programmers try to discourage others from using them because of the distracting number of panes. Though it has a distraction free mode and even without that, if you use it sensibly, you can have it only display the editor and tabs. Pandu Poluan , programmed in Python for nearly a year, to replace complex bash scripts. Answered Mar 24

You don't *have* to use PyCharm, but its features are so good *I* find it essential for Python development.

Things I can't live without:

There are many more PyCharm features, but all the above make PyCharm for me a must-have for Python development.

[Nov 07, 2017] Amazon.com Customer reviews Python Cookbook, Third edition

Notable quotes:
"... There are a couple of quick final points to make about the Python cookbook. Firstly it uses Python 3 and as many very useful third party modules haven't been ported from Python 2.X over to Python 3 yet Python 2.X is still probably still more widely used. ..."
"... this is a language learning book it's not aimed at the novice programmer, think of it more as language nuances and inflections for the experienced Pythonista rather than a how to learn Python book and you won't go far wrong. ..."
"... Most examples are self contained and all the code examples that I tried worked. Additionally, there is a GitHub that the authors created which provides all the code for the examples if you do not want type it yourself. The examples themselves were applied to real world problems; I could see how the recipe was used clearly. When the authors felt they could not provide an entire solution in the text, they point the correct place to visit online. ..."
"... But that's only the beginning. It's hard to describe the pleasure of reading some of the solutions in the Iterators and Generators section, for instance. Actually, I take that back. The pleasure is the same kind as what you may have felt when you first came upon ideas in books such as Bentley's Programming Pearls, way back when. ..."
"... The Active State repository of Python recipes includes many gems, but as the Authors observe in their preference: "most of these recipes are steeped in history and the past". ..."
Nov 07, 2017 | www.amazon.com

renaissance geek on June 23, 2013

The Python Domestic Science Textbook?

A few years ago now I was working in a job that required me to code in PERL. My PERL is passable but no better than that so when I found a copy of the PERL cookbook it was something of a life saver and constant companion. The PERL cookbook is deeply pragmatic and addresses real world problems with the language almost as an afterthought. (Which now I think about is actually a pretty good description of PERL anyway!) The Python cookbook is a very different beast and is much more an exercise in learning the intricacies and nuances of the language. I'm not sure cookbook is the right title - if the PERL Cookbook is a cookbook then the Python Cookbook is more of a domestic science textbook. A bit deeper, a bit dryer and not so focused on immediate problems. This is no way meant to imply that it's a bad book, on the contrary it's a very good book just not entirely what I was expecting.

The book itself is divided into fifteen large sections covering the likes of data structures and algorithms; functions; metaprogramming and concurrency with each section consisting of a number of problems. The problems are structured as a definition of the problem, a solution and a discussion of the solution and how it can be extended. Due to the nature of the Python language a large part of solving the problems lies in knowing which module(s) to include in your code so each problem is generally only a couple of pages, but that is certainly enough to give the solution and reasonably detailed discussion.

As with all books of this type there is going to be some complaints of why is X included and not Y and to be honest if you tried to cover all the possible problems a practicing python programmer is likely to run across the book would end up so large as to be unusable. That being said there was, for me at least, one glaring omission.

I do a lot of data processing with reasonably large data sets, and with the buzz around big data I'm sure I'm not the only one, and frequently find that I have to break down the data sets or I simply consume all the system resources and the program exits. I would have expected at least some treatment of working with very large data sets which seems to be entirely missing.

However this is an issue based on what I use Python for and may very well not matter to you. Even though there may not be exactly the solution you are looking for, there are 260 problems and solutions in the Python cookbook so if you don't learn something new you are probably a certified Python genius and beyond manuals anyway.

There are a couple of quick final points to make about the Python cookbook. Firstly it uses Python 3 and as many very useful third party modules haven't been ported from Python 2.X over to Python 3 yet Python 2.X is still probably still more widely used.

Secondly although this is a language learning book it's not aimed at the novice programmer, think of it more as language nuances and inflections for the experienced Pythonista rather than a how to learn Python book and you won't go far wrong.

Bluegeek on June 13, 2013
Review: "Python Cookbook" by David Beazley and Brian K. Jones; O'Reilly Media

The "Python Cookbook" is a book that brings the Python scripting language to O'Reilly's popular "Cookbook" format. Each Cookbook provides a series of "Recipes" that teach users common techniques that can be used to become productive quickly and as a reference to those who might've forgotten how to do something.

I reviewed this book in the Mobi e-book format. Reading it on Kindle for PC, the Table of Contents only shows the major sections rather than the individual recipes and this made it harder to find what I was looking for. This is apparently a limitation of Kindle for PC, since my Kindle 3 and Kindle for Android had no such issue.

When I use an O'Reilly "Cookbook", I judge it according to its' usefulness: Can I become productive quickly? Is it easy to find what I need? Does it provide helpful tips? Does it teach me where to find the answers to my questions?

This book is not targeted at new Python programmers, but that's where I'm at. The best way for me to learn a new scripting language is to dive right in and try to write something useful, and that was my goal for the "Python Cookbook". I also had "Learning Python" handy to cover any of the basics.

My first Python script was written to read in lists of subnets from two separate files and check that every subnet in list B was also in list A.

I used Recipe 13.3 to parse the command line options. Recipe 5.1 showed me how to read and write files. Recipe 2.11 taught me how to strip carriage returns out of my lines. Recipe 1.10, "Removing Duplicates from a Sequence while Maintaining Order", was very helpful and I was able to reuse the code in my own script. Recipe 2.14, "Combining and Concatenating Strings", helped me with my print statements. Considering this was the first Python script I ever wrote and that it ran, I consider both it and the "Python Cookbook" a success.

I had a bit more trouble with my second script. I was trying to write a script to find the subnet address given an interface address in CIDR notation. Recipe 11.4 introduced the ipaddress module, but this module refused to accept a string variable containing the interface in CIDR notation. I ended up installing another module (netaddr) I found via Google and things went better after that. I suspect the problem was that I was using ActivePython 2.7.2.5 [64 bit] and this book was written for Python 3.

As a DNS professional I was disappointed that there were no DNS-related recipes in the Network and Web Programming section, but Web-related topics were well-represented in the book.

The "Python Cookbook" doesn't seem to have quite the depth and organization of the "Perl Cookbook" but I'm sure I will rely on it heavily as I learn to use Python. It did allow me to be productive very quickly and it passes the "Cookbook" standard with flying colors. Any book that can get me to the point of writing a working, useful script in less than a day is worth using. I recommend this book to anyone who has a basic understanding of Python and wants to get past "Hello, World" and "Eat Spam" as fast as possible.

Reviewer's Note: I received a free copy of the "Python Cookbook" which was used to write this review.

William P Ross Enthusiast: Architecture on May 6, 2016
Treasure Trove of Python Recipes

Python Cookbook goes in depth on a variety of different Python topics. Each section is similar to a question that might be asked on Stack Overflow. The recipes range in difficulty from easy to advanced metaprogramming.

One particular recipe that I liked was 9.1 on how to time a function. When I am using Python I often need to time the code, and usually I need to look up how to do it. This example created a decorator function for timing. It makes it so that you can just put @timethis on top of a function and see how long it takes to execute. I appreciated how elegant this solution was as opposed to the way I was implementing it.

Most examples are self contained and all the code examples that I tried worked. Additionally, there is a GitHub that the authors created which provides all the code for the examples if you do not want type it yourself. The examples themselves were applied to real world problems; I could see how the recipe was used clearly. When the authors felt they could not provide an entire solution in the text, they point the correct place to visit online.

The range in topics was impressive. I found the most challenging chapters to be 9, 12, and 15 which were on metaprogramming, concurrency, and C Extensions. At the beginning of the book the recipes cover topics you would expect like data structures and algorithms, strings, and generators. I found myself surprised that I had not seen a lot of the techniques and solutions before. They were well crafted solutions, and I appreciated how much time and detail the authors must have spent to gather the information.

This is a great reference to have by your side when programming in Python.

Groundhog Day on June 30, 2015
Programming Pearls... Reloaded

Having read some humdrum works in the Cookbook series, my expectations were not very high. However, I soon discovered that this book is in a different league.

When he discusses a problem, Beazley gives you his favorite solution. He also presents alternatives, discusses pros and cons, and calls your attention to subtle details in the solution --- leaving you with a feeling of having learned something of value.

But that's only the beginning. It's hard to describe the pleasure of reading some of the solutions in the Iterators and Generators section, for instance. Actually, I take that back. The pleasure is the same kind as what you may have felt when you first came upon ideas in books such as Bentley's Programming Pearls, way back when.

I hadn't felt that excited about a programming book in a long time. This is one you can take along with you on a weekend just for the pleasure of sipping from it. Sad to say, but there are many O'Reilly books I feel like passing on soon after acquiring them. This one will have a special place on the shelves.

Devendra on September 1, 2013
Extensive tome of recipes for the Python 3 programmer

Python Cookbook is an extensive tome of recipes for the Python 3 programmer. It is a perfect companion book for those migrating Python 2 code to Python 3. If you are stuck with Python 2, you may still find the second edition of the book for sale, but the recipes may be dated as they cover Python 2.4. It is not a beginners book. If you are looking for a beginners book, I recommend Learning Python by Mark Lutz.

A quick chapter summary follows.

I've added this book to my list of references to look into, before heading to Google. Source code listings use syntax highlighting, a nice touch that makes the code easier, and less boring, to read.

I thank O'Reilly media for providing the book for review.

Dan on July 23, 2013
Wisdom - not just examples. Best viewed on a larger screen

The Active State repository of Python recipes includes many gems, but as the Authors observe in their preference: "most of these recipes are steeped in history and the past".

I'd add that the signal to noise ratio seems to be decreasing. The most prolific contributors (with the exception of Raymond Hettinger) have posted trivial examples rather than recipes. This book includes some simple examples too, but it's always in the context of a larger message. Excellent content and advice without the chaff.

I just bought this today. Unlike some early technical Kindle books I've purchased, the formatting is excellent. Kudos to the authors and publisher.

... ... ...

A. Zubarev on September 17, 2013
A book to read and come back again and again

I am tempted to state right away that this book is one of these rare "gems"! Absolutely worth every penny spent and perhaps even more in a way of getting more done in less time or even just can be used to advance professionally. So big thank you to Alex Martelli and David Ascher! I can't imagine how much time, energy, insight and effort the authors put into this book, but it is sure one of the longest professional books I have ever read.

Like I said, this book is very comprehensive at 608 pages long and touches most, if not all, aspects a typical IT pro would deal with in his or her professional life. It may appear though very dry, and in my opinion it should be, but it is the book to come back to again and again, time after time, year after year, so if you need a single specific recipe, you will not feel the book is very short thanks to the way it is structured.

I happen to actually use this book to cope with several assignments at work involving some medium to high complexity data processing for reporting purposes, thus more than a few recipes were used.

Namely, these were "Strings and Text" Ch. 2, "Numbers, Dates and Times" Ch. 3, "Files and I/O" Ch. 4, then hopped to "Functions" Ch. 7, which followed by "Parsing, Modifying and Rewriting XML" Ch. 6.6 and finally landed on "Integrating with a Relational Database" Ch. 6.8. I wish though chapter 7 "Functions" would precede most others because I think it belongs right after "Iterators and generators" which I needed to use as I expanded my program.

I must tell each did its magic, after all Python excels on processing text!

... ... ...

[Nov 07, 2017] Dive Into Python

Nov 07, 2017 | www.diveintopython.net

July 28, 2002

Dive Into Python is a free Python book for experienced programmers. It was originally hosted at DiveIntoPython.org, but the author has pulled down all copies. It is being mirrored here. You can read the book online, or download it in a variety of formats. It is also available in multiple languages . Read Dive Into Python

This book is still being written. You can read the revision history to see what's new. Updated 20 May 2004 . Email me if you'd like to see something changed/updated, or suggestions for this site. Download Dive Into Python

Dive Into Python in your language

Translations are freely permitted as long as they are released under the GNU Free Documentation License. Dive Into Python has already been fully or partially translated into several languages. If you translate it into another language and would like to be listed here, just let me know .

Republish Dive Into Python

Want to mirror this web site? Publish this book on your corporate intranet? Distribute it on CD-ROM ? Feel free. This book is published under the GNU Free Documentation License, which gives you enormous freedoms to modify and redistribute it in all its forms. If you're familiar with the GNU General Public License for software, you already understand these freedoms; the FDL is the GPL for books. You can read the license for all the details.

Copyright © 2000, 2001, 2002, 2003, 2004 Mark Pilgrim
Download Python

Learn Python

[Nov 07, 2017] pdb the Python Debugger

Sep 03, 2017 | docs.python.org
26.2. pdb ! The Python Debugger

Source code: Lib/pdb.py


The module pdb defines an interactive source code debugger for Python programs. It supports setting (conditional) breakpoints and single stepping at the source line level, inspection of stack frames, source code listing, and evaluation of arbitrary Python code in the context of any stack frame. It also supports post-mortem debugging and can be called under program control.

The debugger is extensible ! it is actually defined as the class Pdb . This is currently undocumented but easily understood by reading the source. The extension interface uses the modules bdb and cmd .

The debugger's prompt is (Pdb) . Typical usage to run a program under control of the debugger is:

>>> import pdb

>>> import mymodule

>>> pdb.run('mymodule.test()')

> <string>(0)?()

(Pdb) continue

> <string>(1)?()

(Pdb) continue

NameError: 'spam'

> <string>(1)?()

(Pdb)

pdb.py can also be invoked as a script to debug other scripts. For example:

python -m pdb myscript.py

When invoked as a script, pdb will automatically enter post-mortem debugging if the program being debugged exits abnormally. After post-mortem debugging (or after normal exit of the program), pdb will restart the program. Automatic restarting preserves pdb's state (such as breakpoints) and in most cases is more useful than quitting the debugger upon program's exit.

New in version 2.4: Restarting post-mortem behavior added.

The typical usage to break into the debugger from a running program is to insert

import pdb; pdb.set_trace()

at the location you want to break into the debugger. You can then step through the code following this statement, and continue running without the debugger using the command.

The typical usage to inspect a crashed program is:

>>> import pdb

>>> import mymodule

>>> mymodule.test()

Traceback (most recent call last):

  File "<stdin>", line 1, in <module>

  File "./mymodule.py", line 4, in test

    test2()

  File "./mymodule.py", line 3, in test2

    print spam

NameError: spam

>>> pdb.pm()

> ./mymodule.py(3)test2()

-> print spam

(Pdb)

The module defines the following functions; each enters the debugger in a slightly different way:

pdb. run statement , globals locals ]]
Execute the statement (given as a string) under debugger control. The debugger prompt appears before any code is executed; you can set breakpoints and type continue , or you can step through the statement using step or next (all these commands are explained below). The optional globals and locals arguments specify the environment in which the code is executed; by default the dictionary of the module __main__ is used. (See the explanation of the exec statement or the eval() built-in function.)
pdb. runeval expression , globals locals ]]
Evaluate the expression (given as a string) under debugger control. When runeval() returns, it returns the value of the expression. Otherwise this function is similar to run() .
pdb. runcall function , argument ,
Call the function (a function or method object, not a string) with the given arguments. When runcall() returns, it returns whatever the function call returned. The debugger prompt appears as soon as the function is entered.
pdb. set_trace ()
Enter the debugger at the calling stack frame. This is useful to hard-code a breakpoint at a given point in a program, even if the code is not otherwise being debugged (e.g. when an assertion fails).
pdb. post_mortem traceback
Enter post-mortem debugging of the given traceback object. If no traceback is given, it uses the one of the exception that is currently being handled (an exception must be being handled if the default is to be used).
pdb. pm ()
Enter post-mortem debugging of the traceback found in sys.last_traceback .

The run* functions and set_trace() are aliases for instantiating the Pdb class and calling the method of the same name. If you want to access further features, you have to do this yourself:

class pdb. Pdb completekey='tab' , stdin=None , stdout=None , skip=None
Pdb is the debugger class.

The completekey , stdin and stdout arguments are passed to the underlying cmd.Cmd class; see the description there.

The skip argument, if given, must be an iterable of glob-style module name patterns. The debugger will not step into frames that originate in a module that matches one of these patterns. [1]

Example call to enable tracing with skip :

import pdb; pdb.Pdb(skip=['django.*']).set_trace()

New in version 2.7: The skip argument.
run statement , globals locals ]]
runeval expression , globals locals ]]
runcall function , argument ,
set_trace ()
See the documentation for the functions explained above.
26.3. Debugger Commands

The debugger recognizes the following commands. Most commands can be abbreviated to one or two letters; e.g. h(elp) means that either or help can be used to enter the help command (but not he or hel , nor or Help or HELP ). Arguments to commands must be separated by whitespace (spaces or tabs). Optional arguments are enclosed in square brackets ( [] ) in the command syntax; the square brackets must not be typed. Alternatives in the command syntax are separated by a vertical bar ( ).

Entering a blank line repeats the last command entered. Exception: if the last command was a list command, the next 11 lines are listed.

Commands that the debugger doesn't recognize are assumed to be Python statements and are executed in the context of the program being debugged. Python statements can also be prefixed with an exclamation point ( ). This is a powerful way to inspect the program being debugged; it is even possible to change a variable or call a function. When an exception occurs in such a statement, the exception name is printed but the debugger's state is not changed.

Multiple commands may be entered on a single line, separated by ;; . (A single is not used as it is the separator for multiple commands in a line that is passed to the Python parser.) No intelligence is applied to separating the commands; the input is split at the first ;; pair, even if it is in the middle of a quoted string.

The debugger supports aliases. Aliases can have parameters which allows one a certain level of adaptability to the context under examination.

If a file .pdbrc exists in the user's home directory or in the current directory, it is read in and executed as if it had been typed at the debugger prompt. This is particularly useful for aliases. If both files exist, the one in the home directory is read first and aliases defined there can be overridden by the local file.

h(elp) [ command ]
Without argument, print the list of available commands. With a command as argument, print help about that command. help pdb displays the full documentation file; if the environment variable PAGER is defined, the file is piped through that command instead. Since the command argument must be an identifier, help exec must be entered to get help on the command.
w(here)
Print a stack trace, with the most recent frame at the bottom. An arrow indicates the current frame, which determines the context of most commands.
d(own)
Move the current frame one level down in the stack trace (to a newer frame).
u(p)
Move the current frame one level up in the stack trace (to an older frame).
b(reak) [[ filename :] lineno | function [, condition ]]

With a lineno argument, set a break there in the current file. With a function argument, set a break at the first executable statement within that function. The line number may be prefixed with a filename and a colon, to specify a breakpoint in another file (probably one that hasn't been loaded yet). The file is searched on sys.path . Note that each breakpoint is assigned a number to which all the other breakpoint commands refer.

If a second argument is present, it is an expression which must evaluate to true before the breakpoint is honored.

Without argument, list all breaks, including for each breakpoint, the number of times that breakpoint has been hit, the current ignore count, and the associated condition if any.

tbreak [[ filename :] lineno | function [, condition ]]
Temporary breakpoint, which is removed automatically when it is first hit. The arguments are the same as break.
cl(ear) [ filename:lineno | bpnumber [ bpnumber ]]
With a filename:lineno argument, clear all the breakpoints at this line. With a space separated list of breakpoint numbers, clear those breakpoints. Without argument, clear all breaks (but first ask confirmation).
disable [ bpnumber [ bpnumber ]]
Disables the breakpoints given as a space separated list of breakpoint numbers. Disabling a breakpoint means it cannot cause the program to stop execution, but unlike clearing a breakpoint, it remains in the list of breakpoints and can be (re-)enabled.
enable [ bpnumber [ bpnumber ]]
Enables the breakpoints specified.
ignore bpnumber [ count ]
Sets the ignore count for the given breakpoint number. If count is omitted, the ignore count is set to 0. A breakpoint becomes active when the ignore count is zero. When non-zero, the count is decremented each time the breakpoint is reached and the breakpoint is not disabled and any associated condition evaluates to true.
condition bpnumber [ condition ]
Condition is an expression which must evaluate to true before the breakpoint is honored. If condition is absent, any existing condition is removed; i.e., the breakpoint is made unconditional.
commands [ bpnumber ]

Specify a list of commands for breakpoint number bpnumber . The commands themselves appear on the following lines. Type a line containing just 'end' to terminate the commands. An example:

(Pdb) commands 1

(com) print some_variable

(com) end

(Pdb)

To remove all commands from a breakpoint, type commands and follow it immediately with end; that is, give no commands.

With no bpnumber argument, commands refers to the last breakpoint set.

You can use breakpoint commands to start your program up again. Simply use the continue command, or step, or any other command that resumes execution.

Specifying any command resuming execution (currently continue, step, next, return, jump, quit and their abbreviations) terminates the command list (as if that command was immediately followed by end). This is because any time you resume execution (even with a simple next or step), you may encounter another breakpoint!which could have its own command list, leading to ambiguities about which list to execute.

If you use the 'silent' command in the command list, the usual message about stopping at a breakpoint is not printed. This may be desirable for breakpoints that are to print a specific message and then continue. If none of the other commands print anything, you see no sign that the breakpoint was reached.

New in version 2.5.
s(tep)
Execute the current line, stop at the first possible occasion (either in a function that is called or on the next line in the current function).
n(ext)
Continue execution until the next line in the current function is reached or it returns. (The difference between next and step is that step stops inside a called function, while next executes called functions at (nearly) full speed, only stopping at the next line in the current function.)
unt(il)

Continue execution until the line with the line number greater than the current one is reached or when returning from current frame.

New in version 2.6.
r(eturn)
Continue execution until the current function returns.
c(ont(inue))
Continue execution, only stop when a breakpoint is encountered.
j(ump) lineno

Set the next line that will be executed. Only available in the bottom-most frame. This lets you jump back and execute code again, or jump forward to skip code that you don't want to run.

It should be noted that not all jumps are allowed ! for instance it is not possible to jump into the middle of a for loop or out of a finally clause.

l(ist) [ first [, last ]]
List source code for the current file. Without arguments, list 11 lines around the current line or continue the previous listing. With one argument, list 11 lines around at that line. With two arguments, list the given range; if the second argument is less than the first, it is interpreted as a count.
a(rgs)
Print the argument list of the current function.
p expression

Evaluate the expression in the current context and print its value.

Note

print can also be used, but is not a debugger command ! this executes the Python print statement.

pp expression
Like the command, except the value of the expression is pretty-printed using the pprint module.
alias [ name [command]]

Creates an alias called name that executes command . The command must not be enclosed in quotes. Replaceable parameters can be indicated by %1 , %2 , and so on, while %* is replaced by all the parameters. If no command is given, the current alias for name is shown. If no arguments are given, all aliases are listed.

Aliases may be nested and can contain anything that can be legally typed at the pdb prompt. Note that internal pdb commands can be overridden by aliases. Such a command is then hidden until the alias is removed. Aliasing is recursively applied to the first word of the command line; all other words in the line are left alone.

As an example, here are two useful aliases (especially when placed in the .pdbrc file):

#Print instance variables (usage "pi classInst")

alias pi for k in %1.__dict__.keys(): print "%1.",k,"=",%1.__dict__[k]

#Print instance variables in self

alias ps pi self

unalias name
Deletes the specified alias.
[!] statement

Execute the (one-line) statement in the context of the current stack frame. The exclamation point can be omitted unless the first word of the statement resembles a debugger command. To set a global variable, you can prefix the assignment command with a global command on the same line, e.g.:

(Pdb) global list_options; list_options = ['-l']

(Pdb)

run [ args ]

Restart the debugged Python program. If an argument is supplied, it is split with "shlex" and the result is used as the new sys.argv. History, breakpoints, actions and debugger options are preserved. "restart" is an alias for "run".

New in version 2.6.
q(uit)
Quit from the debugger. The program being executed is aborted.

Footnotes

[1] Whether a frame is considered to originate in a certain module is determined by the __name__ in the frame globals.

[Nov 06, 2017] Dive Deep Into Python Vs Perl Debate - What Should I Learn Python or Perl

Nov 06, 2017 | www.tecmint.com

2. Perl's Built-in Vs Python's 3rd Party Regex and OS Operations Support

Perl language borrows its syntax from and other UNIX commands like sed awk etc. due to which it has way powerful and built in regex support without importing any third-party modules.

Also, Perl can handle OS operations using built-in functions. On the other hand Python has third-party libraries for both the operations i.e. re for regex and os, sys for os operations which need to be ensured before doing such operations.

Perl's regex operations have ' sed ' like syntax which makes it easy not only for search operations but also replace, substitute and other operations on string can be done easily and swiftly than python where a person needs to know and remember the functions which cater to the need.

Example: Consider a program to search for digit in the string in Perl and Python

Python
Import re
str = 'hello0909there'
result = re.findall('\d+',str)
print result
Perl
$string =  'hello0909there';
$string =~ m/(\d+)/;
print "$& \n"

You see the syntax for Perl is way easy and inspired by sed command which takes advantage over Python's syntax which imports third party module 're'

  1. Dominix says: September 26, 2016 at 1:52 pm

    Python centric bullshit

  2. J Cleaver says: September 13, 2016 at 4:40 am

    Some of these perl examples don't really reflect what's more-or-less standard in the Perl community at any time since Perl 5 came out (15 years ago).

    Keeping in mind the vision of TMTOWTDI, your second Perl example:

    open(FILE,"%lt;inp.txt") or die "Can't open file";
    while() {
    print "$_"; }

    really would be typically written as just:

    open (FILE, "inp.txt") or die "Can't open file: $!";
    print while ();

    As many others have pointed out, Perl has a huge amount of syntax flexibility despite its overtones of C heritage, and that allows people to write working code in a relatively ugly, inefficient, and/or hard-to-read manner, with syntax reflecting their experience with other languages.

    It's not really a drawback that Perl is so expressive, but it does mean that the programmer should be as disciplined as the task warrants when writing it when it comes to understandable Perl idioms.

  1. David G. Miller says: September 12, 2016 at 3:04 am

    1) I've usually found that the clarity and elegance of a program have a lot more to do with the programmer than the programming language. People who develop clean solutions will do so regardless of the language of implementation. Likewise, those who can't program will find a way to force an ugly solution out of any language.

    2) Most systems administrators aren't programmers and have rarely had any formal training in software development.

    Put these two observations together and you will still get ugly, "write only" programs Before perl it was shell script, yesterday it was perl, today it's Python. Tomorrow someone will be asking for a replacement for Python because it's so hard to read and can't be maintained. Get used to it (but don't blame the programming language).

    I started my perl programming with perl 2.0 in 1993. It's still my "go to" programming language since it doesn't get in my way and I can get to a solution much faster than with C or shell script.

    • Joe Chakra says: September 12, 2016 at 3:18 am

      Actually performance does matter even for scripting. Imaginew filtering a 100 MB debug log. You could use AWK or gawk, sed or grep but Perl gives a lot more flexibility. Taking 5 seconds is very different to taking ten seconds, just because the more time between request and response the more likely you are to get distracted.

  1. D. B. Dweeb says: September 8, 2016 at 1:30 am

    The Pythonic file handling below surpasses the Perl example, the exception text and file close is automatic. Advantage Python!

    with open("data.csv") as f:
    for line in f:
    print line,

[Nov 06, 2017] Indentation Error in Python - Stack Overflow

Nov 06, 2017 | stackoverflow.com
I can't compile because of this part in my code:
    if command == 'HOWMANY':
        opcodegroupr = "A0"
        opcoder = "85"
    elif command == 'IDENTIFY':
        opcodegroupr = "A0"
        opcoder = "81"

I have this error:

Sorry: IndentationError: ('unindent does not match any outer indentation level', ('wsn.py', 1016, 30, "\t\telif command == 'IDENTIFY':\n"))

But I don't see any indentation error. What can be the problem?

Martijn Pieters ,Feb 20, 2013 at 11:54

You are mixing tabs and spaces.

Find the exact location with:

python -tt yourscript.py

and replace all tabs with spaces. You really want to configure your text editor to only insert spaces for tabs as well.

poke ,Feb 20, 2013 at 11:55

Or the other way around (depends on your personal preference) – poke Feb 20 '13 at 11:55

poke ,Feb 20, 2013 at 12:02

@MartijnPieters If you use tabs, you have tabs, so you do not need to care about its visual presentation. You should never mix tabs and spaces, but apart from that, just choose one and stick to it . You are right, it's a never-ending debate; it totally depends on your personal preference -- hence my comment. – poke Feb 20 '13 at 12:02

neil ,Feb 20, 2013 at 12:02

I have never understood why you would want to use spaces instead of tabs - 1 tab is 1 level of indent and then the size of that is a display preference - but it seems the world disagrees with me. – neil Feb 20 '13 at 12:02

Martijn Pieters ♦ ,Feb 20, 2013 at 12:13

@poke: That's very nice, but in any decent-sized project you will not be the only developer. As soon as you have two people together, there is a large chance you'll disagree about tab size. And pretending that noone will ever make the mistake of mixing tabs and spaces is sticking your head in the sand, frankly. There is a reason that every major style guide for OSS (python or otherwise) states you need to use spaces only . :-) – Martijn Pieters ♦ Feb 20 '13 at 12:13

geoffspear ,Feb 20, 2013 at 12:22

There should be one, and preferably only one, obvious way to do it. Following the style of the python codebase itself is obvious. – geoffspear Feb 20 '13 at 12:22

[Nov 06, 2017] Python Myths about Indentation

Nov 06, 2017 | www.secnetix.de

Python: Myths about Indentation

Note: Lines beginning with " >>> " and " ... " indicate input to Python (these are the default prompts of the interactive interpreter). Everything else is output from Python.

There are quite some prejudices and myths about Python's indentation rules among people who don't really know Python. I'll try to address a few of these concerns on this page.


"Whitespace is significant in Python source code."

No, not in general. Only the indentation level of your statements is significant (i.e. the whitespace at the very left of your statements). Everywhere else, whitespace is not significant and can be used as you like, just like in any other language. You can also insert empty lines that contain nothing (or only arbitrary whitespace) anywhere.

Also, the exact amount of indentation doesn't matter at all, but only the relative indentation of nested blocks (relative to each other).

Furthermore, the indentation level is ignored when you use explicit or implicit continuation lines. For example, you can split a list across multiple lines, and the indentation is completely insignificant. So, if you want, you can do things like this:

>>> foo = [
... 'some string',
... 'another string',
... 'short string'
... ]
>>> print foo
['some string', 'another string', 'short string']

>>> bar = 'this is ' \
... 'one long string ' \
... 'that is split ' \
... 'across multiple lines'
>>> print bar
this is one long string that is split across multiple lines

"Python forces me to use a certain indentation style."

Yes and no. First of all, you can write the inner block all on one line if you like, therefore not having to care about intendation at all. The following three versions of an "if" statement are all valid and do exactly the same thing (output omitted for brevity):

>>> if 1 + 1 == 2:
... print "foo"
... print "bar"
... x = 42

>>> if 1 + 1 == 2:
... print "foo"; print "bar"; x = 42

>>> if 1 + 1 == 2: print "foo"; print "bar"; x = 42

Of course, most of the time you will want to write the blocks in separate lines (like the first version above), but sometimes you have a bunch of similar "if" statements which can be conveniently written on one line each.

If you decide to write the block on separate lines, then yes, Python forces you to obey its indentation rules, which simply means: The enclosed block (that's two "print" statements and one assignment in the above example) have to be indented more than the "if" statement itself. That's it. And frankly, would you really want to indent it in any other way? I don't think so.

So the conclusion is: Python forces you to use indentation that you would have used anyway, unless you wanted to obfuscate the structure of the program. In other words: Python does not allow to obfuscate the structure of a program by using bogus indentations. In my opinion, that's a very good thing.

Have you ever seen code like this in C or C++?

/* Warning: bogus C code! */

if (some condition)
if (another condition)
do_something(fancy);
else
this_sucks(badluck);

Either the indentation is wrong, or the program is buggy, because an "else" always applies to the nearest "if", unless you use braces. This is an essential problem in C and C++. Of course, you could resort to always use braces, no matter what, but that's tiresome and bloats the source code, and it doesn't prevent you from accidentally obfuscating the code by still having the wrong indentation. (And that's just a very simple example. In practice, C code can be much more complex.)

In Python, the above problems can never occur, because indentation levels and logical block structure are always consistent. The program always does what you expect when you look at the indentation.

Quoting the famous book writer Bruce Eckel:

Because blocks are denoted by indentation in Python, indentation is uniform in Python programs. And indentation is meaningful to us as readers. So because we have consistent code formatting, I can read somebody else's code and I'm not constantly tripping over, "Oh, I see. They're putting their curly braces here or there." I don't have to think about that.


"You cannot safely mix tabs and spaces in Python."

That's right, and you don't want that. To be exact, you cannot safely mix tabs and spaces in C either: While it doesn't make a difference to the compiler, it can make a big difference to humans looking at the code. If you move a piece of C source to an editor with different tabstops, it will all look wrong (and possibly behave differently than it looks at first sight). You can easily introduce well-hidden bugs in code that has been mangled that way. That's why mixing tabs and spaces in C isn't really "safe" either. Also see the "bogus C code" example above.

Therefore, it is generally a good idea not to mix tabs and spaces for indentation. If you use tabs only or spaces only, you're fine.

Furthermore, it can be a good idea to avoid tabs alltogether, because the semantics of tabs are not very well-defined in the computer world, and they can be displayed completely differently on different types of systems and editors. Also, tabs often get destroyed or wrongly converted during copy&paste operations, or when a piece of source code is inserted into a web page or other kind of markup code.

Most good editors support transparent translation of tabs, automatic indent and dedent. That is, when you press the tab key, the editor will insert enough spaces (not actual tab characters!) to get you to the next position which is a multiple of eight (or four, or whatever you prefer), and some other key (usually Backspace) will get you back to the previous indentation level.

In other words, it's behaving like you would expect a tab key to do, but still maintaining portability by using spaces in the file only. This is convenient and safe.

Having said that -- If you know what you're doing, you can of course use tabs and spaces to your liking, and then use tools like "expand" (on UNIX machines, for example) before giving the source to others. If you use tab characters, Python assumes that tab stops are eight positions apart.


"I just don't like it."

That's perfectly OK; you're free to dislike it (and you're probably not alone). Granted, the fact that indentation is used to indicate the block structure might be regarded as uncommon and requiring to get used to it, but it does have a lot of advantages, and you get used to it very quickly when you seriously start programming in Python.

Having said that, you can use keywords to indicate the end of a block (instead of indentation), such as " endif ". These are not really Python keywords, but there is a tool that comes with Python which converts code using "end" keywords to correct indentation and removes those keywords. It can be used as a pre-processor to the Python compiler. However, no real Python programmer uses it, of course.
[Update] It seems this tool has been removed from recent versions of Python. Probably because nobody really used it.


"How does the compiler parse the indentation?"

The parsing is well-defined and quite simple. Basically, changes to the indentation level are inserted as tokens into the token stream.

The lexical analyzer (tokenizer) uses a stack to store indentation levels. At the beginning, the stack contains just the value 0, which is the leftmost position. Whenever a nested block begins, the new indentation level is pushed on the stack, and an "INDENT" token is inserted into the token stream which is passed to the parser. There can never be more than one "INDENT" token in a row.

When a line is encountered with a smaller indentation level, values are popped from the stack until a value is on top which is equal to the new indentation level (if none is found, a syntax error occurs). For each value popped, a "DEDENT" token is generated. Obviously, there can be multiple "DEDENT" tokens in a row.

At the end of the source code, "DEDENT" tokens are generated for each indentation level left on the stack, until just the 0 is left.

Look at the following piece of sample code:

>>> if foo:
... if bar:
... x = 42
... else:
... print foo
...
In the following table, you can see the tokens produced on the left, and the indentation stack on the right.
<if> <foo> <:> [0]
<INDENT> <if> <bar> <:> [0, 4]
<INDENT> <x> <=> <42> [0, 4, 8]
<DEDENT> <DEDENT> <else> <:> [0]
<INDENT> <print> <foo> [0, 2]
<DEDENT> [0]
Note that after the lexical analysis (before parsing starts), there is no whitespace left in the list of tokens (except possibly within string literals, of course). In other words, the indentation is handled by the lexer, not by the parser.

The parser then simply handles the "INDENT" and "DEDENT" tokens as block delimiters -- exactly like curly braces are handled by a C compiler.

The above example is intentionally simple. There are more things to it, such as continuation lines. They are well-defined, too, and you can read about them in the Python Language Reference if you're interested, which includes a complete formal grammar of the language.

[Nov 06, 2017] Modular Programming with Python by Erik Westra

Nov 06, 2017 | www.amazon.com
  • Paperback: 246 pages
  • Publisher: Packt Publishing - ebooks Account (May 26, 2016)
  • Language: English
  • ISBN-10: 1785884484
  • ISBN-13: 978-1785884481
  • Product Dimensions: 7.5 x 0.6 x 9.2 inches

Contents:

Modular Programming with Python

1. Introducing Modular Programming

2. Writing Your First Modular Program

3. Using Modules and Packages

4. Using Modules for Real-World Programming

5. Working with Module Patterns

6. Creating Reusable Modules

7. Advanced Module Techniques

8. Testing and Deploying Modules

9. Modular Programming as a Foundation for Good Programming Technique

By kievite on November 5, 2017

Great book on a very important topic. Highly recommended

Great book on a very important topic.

Python is complex language with even more complex environment and its module system is the critical part of it. For example Python standard library is structured as a collection of modules. The author gives you an excellent overview of Python module system, gives important recommendation about creation of your own modules (and provide several examples, including generator example in chapter 4, as well as warns about gotchas. Interplay between modules and namespaces covered in Chapter 7 is alone worth several times of the price of the book. for example few understand that the import statement adds the imported module or package to the current namespace, which may or may not be the global namespace. the author also covers the problem of "name masking" in this chapter.

Ability to write a large script using your own modules is a very important skill that few books teach. usually intro books on Python try to throw everything that language contains into the sink, creating problems for whose who study the language, even in cases when they already knew some other programming languages such as C++ or Perl. Ability not to cover some features of the language usually are complete absent in authors of such books.

Most of the authors of Python books talks a lot about how great Python is, but never explain why. this books explains probably the most important feature of this scripting language which makes is great (actually inhered from Modula 3). Also most intro books suffer from excessive fascination with OO (thanks God this fad is past its peak). This book does not.

Publishing of books that are devoted to important topics has great value as:you have nowhere to go to get information that it provides. But it is very risky business. Of cause if you are diligent you can collect this information by reading a dozen of book by extracting and organizing into some presentation relevant parts. But this is the work better reserved for personalities which corresponds to famous Sherlock Holms and it presuppose that you have pretty of time to do it. Usually meeting both of those two conditions is pretty unrealistic.

So it takes a certain about of courage to write a book devoted to a single specific feature of Python and the author should be commended for that.

That's why I highly recommend this book for anybody who is trying to learn the language. It really allow you to understand a single the most critical feature of the Python language.

The book contain 9 chapters. Here are the titles of those chapters:

1. Introducing Modular Programming
2. Writing Your First Modular Program
3. Using Modules and Packages
4. Using Modules for Real-World Programming
5. Working with Module Patterns
6. Creating Reusable Modules
7. Advanced Module Techniques
8. Testing and Deploying Modules
9. Modular Programming as a Foundation for Good Programming Technique

NOTE: In chapter 8 the author covers unrelated but an important topic about how to prepare your modules to publication and upload them to GitHub. Using GitHub became now very popular among Python programmers and the earlier you learn about this possibility the better.

Chapter 8 also covers important topic about installation of Python packages. But unfortunately the coverage is way to brief and does not cover gotchas that you might experience installing such packages as Numpy.

I would like to stress it again: currently the book has no competition in the level of coverage of this, probably the most important feature of Python language.

[Nov 04, 2017] Which is the best book for learning python for absolute beginners on their own?

Nov 04, 2017 | www.quora.com

Robert Love Software Engineer at Google

Mark Lutz's Learning Python is a favorite of many. It is a good book for novice programmers. The new fifth edition is updated to both Python 2.7 and 3.3. Thank you for your feedback! Your response is private. Is this answer still relevant and up to date?

Aditi Sharma , i love coding Answered Jul 10 2016

Originally Answered: Which is the best book for learning Python from beginners to advanced level?

Instead of book, I would advice you to start learning Python from CodesDope which is a wonderful site for starting to learn Python from the absolute beginning. The way its content explains everything step-by-step and in such an amazing typography that makes learning just fun and much more easy. It also provides you with a number of practice questions for each topic so that you can make your topic even stronger by solving its questions just after reading it and you won't have to go around searching for its questions for practice. Moreover, it has a discussion forum which is really very responsive in solving all your doubts instantly.

3.1k Views 11 Upvotes Promoted by Facebook Join Facebook Engineering Leadership. We're hiring! Join our engineering leadership team to help us bring the world closer together. Learn More at facebook.com/careers Alex Forsyth , Computer science major at MIT Answered Dec 28 2015 Originally Answered: What is the best way to learn to code? Specifically Python.

There are many good websites for learning the basics, but for going a bit deeper, I'd suggest MIT OCW 6.00SC. This is how I learned Python back in 2012 and what ultimately led me to MIT and to major in CS. 6.00 teaches Python syntax but also teaches some basic computer science concepts. There are lectures from John Guttag, which are generally well done and easy to follow. It also provides access to some of the assignments from that semester, which I found extremely useful in actually learning Python.

After completing that, you'd probably have a better idea of what direction you wanted to go. Some examples could be completing further OCW courses or completing projects in Python.

[Sep 18, 2017] Operators and String Formatting in Python Operators

Sep 18, 2017 | www.informit.com

InformIT

Formatting Strings!Modulus

Although not actually modulus, the Python % operator works similarly in string formatting to interpolate variables into a formatting string. If you've programmed in C, you'll notice that % is much like C's printf(), sprintf(), and fprintf() functions.

There are two forms of %, one of which works with strings and tuples, the other with dictionaries.

StringOperand % TupleOperand 

StringOperand % DictionaryOperand

Both return a new formatted string quickly and easily.

% Tuple String Formatting

In the StringOperand % TupleOperand form, StringOperand represents special directives within the string that help format the tuple. One such directive is %s, which sets up the format string

>>> format = "%s is my friend and %s is %s years old"

and creates two tuples, Ross_Info and Rachael_Info.

>>> Ross_Info = ("Ross", "he", 28)

>>> Rachael_Info = ("Rachael", "she", 28)

The format string operator (%) can be used within a print statement, where you can see that every occurrence of %s is respectively replaced by the items in the tuple.

>>> print (format % Ross_Info) 

Ross is my friend and he is 28 years old 



>>> print (format % Rachael_Info) 

Rachael is my friend and she is 28 years old

Also note that %s automatically converts the last item in the tuple to a reasonable string representation. Here's an example of how it does this using a list:

>>> bowling_scores = [190, 135, 110, 95, 195]

>>> name = "Ross"

>>> strScores = "%s's bowling scores were %s" \

...                                                 % (name, bowling_scores) 

>>> print strScores 

Ross's bowling scores were [190, 135, 110, 95, 195]

First, we create a list variable called bowling_scores and then a string variable called name. We then use a string literal for a format string (StringOperand) and use a tuple containing name and bowling_scores.

Format Directives

Table 3–6 covers all of the format directives and provides a short example of usage for each. Note that the tuple argument containing a single item can be denoted with the % operator as item, or (item).

Table 3–6 Format Directives
Directive Description Interactive Session
%s Represents a value as a string >>> list = ["hi", 1, 1.0, 1L]
>>> "%s" % list
"['hi', 1, 1.0, 1L]"
>>> "list equals %s" % list
"list equals ['hi', 1, 1.0, 1L]"
%i Integer >>> "i = %i" % (5)
'i = 5'
>>> "i = %3i" % (5)
'i = 5'
%d Decimal integer >>> "d = %d" % 5
'd = 5'
>>> "%3d" % (3)
' 3'
%x Hexadecimal integer >>> "%x" % (0xff)
'ff'
>>> "%x" % (255)
'ff'
%x Hexadecimal integer >>> "%x" % (0xff)
'ff'
>>> "%x" % (255)
'ff'
%o Octal integer >>> "%o" % (255)
377
>>> "%o" % (0377)
377
%u Unsigned integer >>> print "%u" % -2000
2147481648
>>> print "%u" % 2000
2000
%e Float exponent >>> print "%e" % (30000000L)
3.000000e+007
>>> "%5.2e" % (300000000L)
'3.00e+008'
%f Float >>> "check = %1.2f" % (3000)
'check = 3000.00'
>>> "payment = $%1.2f" % 3000
'payment = $3000.00'
%g Float exponent >>> "%3.3g" % 100
'100.'
>>> "%3.3g" % 1000000000000L
'10.e11'
>>> "%g" % 100
'100.'
%c ASCII character >>> "%c" % (97)
'a'
>>> "%c" % 97
'a'
>>> "%c" % (97)
'a'

Table 3–7 shows how flags can be used with the format directives to add leading zeroes or spaces to a formatted number. They should be inserted immediately after the %.

Table 3–7 Format Directive Flags
Flag Description Interactive Session
# Forces octal to have a 0 prefix; forces hex to >>> "%#x" % 0xff
have a 0x prefix '0xff'
>>> "%#o" % 0377
'0ff'
+ Forces a positive number to have a sign >>> "%+d" % 100
'+100'
- Left justification (default is right) >>> "%-5d, %-5d" % (10,10)
'10 , 10 '
" " Precedes a positive number with a blank space >>> "% d,% d" % (-10, 10)
'-100,10'
0 0 padding instead of spaces >>> "%05d" % (100,)
'00100'

Advanced Topic: Using the %d, %i, %f, and %e Directives for Formatting Numbers

The % directives format numeric types: %i works with Integer; %f and %e work with Float with and without scientific notation, respectively.

>>> "%i, %f, %e" % (1000, 1000, 1000) 

'1000, 1000.000000, 10.000000e+002'

Notice how awkward all of those zeroes look. You can limit the length of precision and neaten up your code like this:

>>> "%i, %2.2f, %2.2e" % (1000, 1000, 1000) 

'1000, 1000.00, 10.00e+002'

The %2.2f directive tells Python to format the number as at least two characters and to cut the precision to two characters after the decimal point. This is useful for printing floating-point numbers that represent currency.

>>> "Your monthly payments are $%1.2f" % (payment) 

'Your monthly payments are $444.43'

All % directives have the form %min.precision(type), where min is the minimum length of the field, precision is the length of the mantissa (the numbers on the right side of the decimal point), and type is the type of directive (e, f, i, or d). If the precision field is missing, the directive can take the form %min(type), so, for example, %5d ensures that a decimal number has at least 5 fields and %20f ensures that a floating-point number has at least 20.

Let's look at the use of these directives in an interactive session.

>>> "%5d" % (100,) 

' 100' 

>>> "%20f" % (100,) 

' 100.000000'

Here's how to truncate the float's mantissa to 2 with %20.2f.

>>> "%20.2f" % (100,) 

' 100.00'

The padding that precedes the directive is useful for printing rows and columns of data for reporting because it makes the printed output easy to read. This can be seen in the following example (from format.py ):

     # Create two rows

row1 = (100, 10000, 20000, 50000, 6000, 6, 5) 

row2 = (1.0, 2L, 5, 2000, 56, 6.0, 7) 



      # 

      # Print out the rows without formatting 

print "here is an example of the columns not lining up" 

print ´row1´ + "\n" + ´row2´ 

print 

      # 

      # Create a format string that forces the number

      # to be at least 3 characters long to the left

      # and 2 characters to the right of the decimal point

format = "(%3.2e, %3.2e, %3.2e, %3.2e, " + \ "%3.2e, %3.2e, %3.2e)" 



      # 

      # Create a string for both rows

      # using the format operator

strRow1 = format % row1 

strRow2 = format % row2 

print "here is an example of the columns" + \ 

        " lining up using \%e" 



print strRow1 + "\n" + strRow2 

print 



      # Do it again this time with the %i and %d directive 

format1 = "(%6i, %6i, %6i, %6i, %6i, %6i, %6i)" 

format2 = "(%6d, %6d, %6d, %6d, %6d, %6d, %6d)" 

strRow1 = format1 % row1 

strRow2 = format2 % row2 

print "here is an example of the columns" + \ 

        " lining up using \%i and \%d" 



print strRow1 + "\n" + strRow2 

print 



here is an example of the columns not lining up 

(100, 10000, 20000, 50000, 6000, 6, 5) 

(1.0, 2L, 5, 2000, 56, 6.0, 7) 



here is an example of the columns lining up using \%e 

(1.00e+002, 1.00e+004, 2.00e+004, 5.00e+004, 6.00e+003, 6.00e+000, 5.00e+000) 

(1.00e+000, 2.00e+000, 5.00e+000, 2.00e+003, 5.60e+001, 6.00e+000, 7.00e+000) 



here is an example of the columns lining up using \%i and \%d 

( 100, 10000, 20000, 50000, 6000, 6, 5) 

(     1,         2,         5,   2000,     56, 6, 7)

You can see that the %3.2e directive permits a number to take up only three spaces plus the exponential whereas %6d and %6i permit at least six spaces. Note that %i and %d do the same thing that %e does. Most C programmers are familiar with %d but may not be familiar with %i, which is a recent addition to that language.

String % Dictionary

Another useful Python feature for formatting strings is StringOperand % Dictio-naryOperand. This form allows you to customize and print named fields in the string. %(Income)d formats the value referenced by the Income key. Say, for example, that you have a dictionary like the one here:

Monica = { 

                 "Occupation": "Chef",

                 "Name" : "Monica", 

                 "Dating" : "Chandler",

                 "Income" : 40000 

                  }

With %(Income)d, this is expressed as

>>> "%(Income)d" % Monica 

'40000'

Now let's say you have three best friends, whom you define as dictionaries named Monica, Chandler, and Ross.

Monica = { 

                 "Occupation": "Chef",

                 "Name" : "Monica", 

                 "Dating" : "Chandler", 

                 "Income" : 40000 

                 } 



Ross = 

                               { 

                "Occupation": "Scientist Museum Dude",

                "Name" : "Ross", 

                "Dating" : "Rachael", 

                "Income" : 70000 

                } 



Chandler =              { 

                "Occupation": "Buyer",

                "Name" : "Chandler", 

                "Dating" : "Monica", 

                "Income" : 65000 

                }

To write them a form letter, you can create a format string called message that uses all of the above dictionaries' keywords.

message = "%(Name)s, %(Occupation)s, %(Dating)s," \ 

                  " %(Income)2.2f"

Notice that %(Income)2.2f formats this with a floating-point precision of 2, which is good for currency. The output is

Chandler, Buyer, Monica, 65000.00 

Ross, Scientist Museum Dude, Rachael, 70000.00 

Monica, Chef, Chandler, 40000.00

You can then print each dictionary using the format string operator.

print message % Chandler 

print message % Ross 

print message % Monica

To generate your form letter and print it out to the screen, you first create a format string called dialog.

dialog = """ 

Hi %(Name)s, 



How are you doing? How is %(Dating)s? 

Are you still seeing %(Dating)s? 



How is work at the office? 

I bet it is hard being a %(Occupation)s. 

I know I could not do it. 

"""

Then you print out each dictionary using the dialog format string with the % format string operator.

print dialog % Ross 

print dialog % Chandler 

print dialog % Monica

The output is

Hi Ross, 



How are you doing? How is Rachael? 

Are you still seeing Rachael? 



How is work at the office? 

I bet it is hard being a Scientist Museum Dude. 

I know I could not do it. 



Hi Chandler, 

How are you doing? How is Monica? 

Are you still seeing Monica? 



How is work at the office? 

I bet it is hard being a Buyer. 

I know I could not do it. 



Hi Monica, 

How are you doing? How is Chandler? 

Are you still seeing Chandler? 



How is work at the office? 

I bet it is hard being a Chef. 

I know I could not do it.

%(Income)d is a useful, flexible feature. You just saw how much time it can save you in writing form letters. Imagine what it can do for writing reports. < Back Page

[Sep 16, 2017] Is Python Really the Fastest-Growing Programming Language?

Sep 16, 2017 | developers.slashdot.org

(stackoverflow.blog) 253 Posted by EditorDavid on Saturday September 09, 2017 @09:10PM from the is-simple-better-than-complex? dept. An anonymous reader quotes Stack Overflow Blog: In this post, we'll explore the extraordinary growth of the Python programming language in the last five years, as seen by Stack Overflow traffic within high-income countries.

The term "fastest-growing" can be hard to define precisely, but we make the case that Python has a solid claim to being the fastest-growing major programming language ... June 2017 was the first month that Python was the most visited [programming language] tag on Stack Overflow within high-income nations. This included being the most visited tag within the US and the UK, and in the top 2 in almost all other high income nations (next to either Java or JavaScript). This is especially impressive because in 2012, it was less visited than any of the other 5 languages, and has grown by 2.5-fold in that time .

Part of this is because of the seasonal nature of traffic to Java. Since it's heavily taught in undergraduate courses, Java traffic tends to rise during the fall and spring and drop during the summer. Does Python show a similar growth in the rest of the world, in countries like India, Brazil, Russia and China? Indeed it does.

Outside of high-income countries Python is still the fastest growing major programming language; it simply started at a lower level and the growth began two years later (in 2014 rather than 2012). In fact, the year-over-year growth rate of Python in non-high-income countries is slightly higher than it is in high-income countries ...

We're not looking to contribute to any "language war." The number of users of a language doesn't imply anything about its quality, and certainly can't tell you which language is more appropriate for a particular situation.

With that perspective in mind, however, we believe it's worth understanding what languages make up the developer ecosystem, and how that ecosystem might be changing. This post demonstrated that Python has shown a surprising growth in the last five years, especially within high-income countries.

The post was written by Stack Overflow data scientist David Robinson, who notes that "I used to program primarily in Python, though I have since switched entirely to R."

[Sep 03, 2017] Python Conquers The Universe

Notable quotes:
"... If you press ENTER without entering anything, pdb will re-execute the last command that you gave it. ..."
"... When you use "s" to step into subroutines, you will often find yourself trapped in a subroutine. You have examined the code that you're interested in, but now you have to step through a lot of uninteresting code in the subroutine. ..."
"... In this situation, what you'd like to be able to do is just to skip ahead to the end of the subroutine. That is, you want to do something like the "c" ("continue") command does, but you want just to continue to the end of the subroutine, and then resume your stepping through the code. ..."
"... You can do it. The command to do it is "r" (for "return" or, better, "continue until return"). If you are in a subroutine and you enter the "r" command at the (Pdb) prompt, pdb will continue executing until the end of the subroutine. At that point ! the point when it is ready to return to the calling routine ! it will stop and show the (Pdb) prompt again, and you can resume stepping through your code. ..."
developers.slashdot.org
Debugging in Python Posted on 2009/09/10 by Steve Ferg As a programmer, one of the first things that you need for serious program development is a debugger.

Python has a debugger, which is available as a module called pdb (for "Python DeBugger", naturally!). Unfortunately, most discussions of pdb are not very useful to a Python newbie ! most are very terse and simply rehash the description of pdb in the Python library reference manual . The discussion that I have found most accessible is in the first four pages of Chapter 27 of the Python 2.1 Bible .

So here is my own personal gentle introduction to using pdb. It assumes that you are not using any IDE ! that you're coding Python with a text editor and running your Python programs from the command line.

Some Other Debugger Resources Getting started ! pdb.set_trace()

To start, I'll show you the very simplest way to use the Python debugger.

1. Let's start with a simple program, epdb1.py.

# epdb1.py -- experiment with the Python debugger, pdb

a = "aaa"

b = "bbb"

c = "ccc"

final = a + b + c

print final

2. Insert the following statement at the beginning of your Python program. This statement imports the Python debugger module, pdb.

import pdb

3. Now find a spot where you would like tracing to begin, and insert the following code:

pdb.set_trace()

So now your program looks like this.

# epdb1.py -- experiment with the Python debugger, pdb

import pdb

a = "aaa"

pdb.set_trace()

b = "bbb"

c = "ccc"

final = a + b + c

print final

4. Now run your program from the command line as you usually do, which will probably look something like this:

PROMPT> python epdb1.py

When your program encounters the line with pdb.set_trace() it will start tracing. That is, it will (1) stop, (2) display the "current statement" (that is, the line that will execute next) and (3) wait for your input. You will see the pdb prompt, which looks like this:

(Pdb)
Execute the next statement with "n" (next)

At the (Pdb) prompt, press the lower-case letter "n" (for "next") on your keyboard, and then press the ENTER key. This will tell pdb to execute the current statement. Keep doing this ! pressing "n", then ENTER.

Eventually you will come to the end of your program, and it will terminate and return you to the normal command prompt.

Congratulations! You've just done your first debugging run!

Repeating the last debugging command with ENTER

This time, do the same thing as you did before. Start your program running. At the (Pdb) prompt, press the lower-case letter "n" (for "next") on your keyboard, and then press the ENTER key.

But this time, after the first time that you press "n" and then ENTER, don't do it any more. Instead, when you see the (Pdb) prompt, just press ENTER. You will notice that pdb continues, just as if you had pressed "n". So this is Handy Tip #1:

If you press ENTER without entering anything, pdb will re-execute the last command that you gave it.

In this case, the command was "n", so you could just keep stepping through the program by pressing ENTER.

Notice that as you passed the last line (the line with the "print" statement), it was executed and you saw the output of the print statement ("aaabbbccc") displayed on your screen.

Quitting it all with "q" (quit)

The debugger can do all sorts of things, some of which you may find totally mystifying. So the most important thing to learn now ! before you learn anything else ! is how to quit debugging!

It is easy. When you see the (Pdb) prompt, just press "q" (for "quit") and the ENTER key. Pdb will quit and you will be back at your command prompt. Try it, and see how it works.

Printing the value of variables with "p" (print)

The most useful thing you can do at the (Pdb) prompt is to print the value of a variable. Here's how to do it.

When you see the (Pdb) prompt, enter "p" (for "print") followed by the name of the variable you want to print. And of course, you end by pressing the ENTER key.

Note that you can print multiple variables, by separating their names with commas (just as in a regular Python "print" statement). For example, you can print the value of the variables a, b, and c this way:

p a, b, c
When does pdb display a line?

Suppose you have progressed through the program until you see the line

final = a + b + c

and you give pdb the command

p final

You will get a NameError exception. This is because, although you are seeing the line, it has not yet executed. So the final variable has not yet been created.

Now press "n" and ENTER to continue and execute the line. Then try the "p final" command again. This time, when you give the command "p final", pdb will print the value of final , which is "aaabbbccc".

Turning off the (Pdb) prompt with "c" (continue)

You probably noticed that the "q" command got you out of pdb in a very crude way ! basically, by crashing the program.

If you wish simply to stop debugging, but to let the program continue running, then you want to use the "c" (for "continue") command at the (Pdb) prompt. This will cause your program to continue running normally, without pausing for debugging. It may run to completion. Or, if the pdb.set_trace() statement was inside a loop, you may encounter it again, and the (Pdb) debugging prompt will appear once more.

Seeing where you are with "l" (list)

As you are debugging, there is a lot of stuff being written to the screen, and it gets really hard to get a feeling for where you are in your program. That's where the "l" (for "list") command comes in. (Note that it is a lower-case "L", not the numeral "one" or the capital letter "I".)

"l" shows you, on the screen, the general area of your program's souce code that you are executing. By default, it lists 11 (eleven) lines of code. The line of code that you are about to execute (the "current line") is right in the middle, and there is a little arrow "–>" that points to it.

So a typical interaction with pdb might go like this

Stepping into subroutines with "s" (step into)

Eventually, you will need to debug larger programs ! programs that use subroutines. And sometimes, the problem that you're trying to find will lie buried in a subroutine. Consider the following program.

# epdb2.py -- experiment with the Python debugger, pdb

import pdb



def combine(s1,s2):      # define subroutine combine, which...

    s3 = s1 + s2 + s1    # sandwiches s2 between copies of s1, ...

    s3 = '"' + s3 +'"'   # encloses it in double quotes,...

    return s3            # and returns it.



a = "aaa"

pdb.set_trace()

b = "bbb"

c = "ccc"

final = combine(a,b)

print final

As you move through your programs by using the "n" command at the (Pdb) prompt, you will find that when you encounter a statement that invokes a subroutine ! the final = combine(a,b) statement, for example ! pdb treats it no differently than any other statement. That is, the statement is executed and you move on to the next statement ! in this case, to print final.

But suppose you suspect that there is a problem in a subroutine. In our case, suppose you suspect that there is a problem in the combine subroutine. What you want ! when you encounter the final = combine(a,b) statement ! is some way to step into the combine subroutine, and to continue your debugging inside it.

Well, you can do that too. Do it with the "s" (for "step into") command.

When you execute statements that do not involve function calls, "n" and "s" do the same thing ! move on to the next statement. But when you execute statements that invoke functions, "s", unlike "n", will step into the subroutine. In our case, if you executed the

final = combine(a,b)

statement using "s", then the next statement that pdb would show you would be the first statement in the combine subroutine:

def combine(s1,s2):

and you will continue debugging from there.

Continuing but just to the end of the current subroutine with "r" (return)

When you use "s" to step into subroutines, you will often find yourself trapped in a subroutine. You have examined the code that you're interested in, but now you have to step through a lot of uninteresting code in the subroutine.

In this situation, what you'd like to be able to do is just to skip ahead to the end of the subroutine. That is, you want to do something like the "c" ("continue") command does, but you want just to continue to the end of the subroutine, and then resume your stepping through the code.

You can do it. The command to do it is "r" (for "return" or, better, "continue until return"). If you are in a subroutine and you enter the "r" command at the (Pdb) prompt, pdb will continue executing until the end of the subroutine. At that point ! the point when it is ready to return to the calling routine ! it will stop and show the (Pdb) prompt again, and you can resume stepping through your code.

You can do anything at all at the (Pdb) prompt

Sometimes you will be in the following situation ! You think you've discovered the problem. The statement that was assigning a value of, say, "aaa" to variable var1 was wrong, and was causing your program to blow up. It should have been assigning the value "bbb" to var1.

at least, you're pretty sure that was the problem

What you'd really like to be able to do, now that you've located the problem, is to assign "bbb" to var1, and see if your program now runs to completion without bombing.

It can be done!

One of the nice things about the (Pdb) prompt is that you can do anything at it ! you can enter any command that you like at the (Pdb) prompt. So you can, for instance, enter this command at the (Pdb) prompt.

(Pdb) var1 = "bbb"

You can then continue to step through the program. Or you could be adventurous ! use "c" to turn off debugging, and see if your program will end without bombing!

but be a little careful!

[Thanks to Dick Morris for the information in this section.]

Since you can do anything at all at the (Pdb) prompt, you might decide to try setting the variable b to a new value, say "BBB", this way:

(Pdb) b = "BBB"

If you do, pdb produces a strange error message about being unable to find an object named '= "BBB" '. Why???

What happens is that pdb attempts to execute the pdb command for setting and listing breakpoints (a command that we haven't discussed). It interprets the rest of the line as an argument to the command, and can't find the object that (it thinks) is being referred to. So it produces an error message.

So how can we assign a new value to ? The trick is to start the command with an exclamation point (!).

(Pdb)!b = "BBB"

An exclamation point tells pdb that what follows is a Python statement, not a pdb command.

The End

Well, that's all for now. There are a number of topics that I haven't mentioned, such as help, aliases, and breakpoints. For information about them, try the online reference for pdb commands on the Python documentation web site. In addition, I recommend Jeremy Jones' article Interactive Debugging in Python in O'Reilly's Python DevCenter.

I hope that this introduction to pdb has been enough to get you up and running fairly quickly and painlessly. Good luck!

!Steve Ferg

Sep 03, 2017 | pythonconquerstheuniverse.wordpress.com

[Dec 26, 2016] PyCharm - The Best Linux Python IDE

Dec 26, 2016 | www.lifewire.com
by Gary Newell Updated September 23, 2016 Introduction

In this guide I will introduce you to the PyCharm integrated development environment which can be used to develop professional applications using the Python programming language.

Python is a great programming language because it is truly cross platform and can be used to develop a single application which will run on Windows, Linux and Mac computers without having to recompile any code.

PyCharm is an editor and debugger developed by Jetbrains who are the same people who developed Resharper which is a great tool used by Windows developers for refactoring code and to make their lives easier when writing .NET code. Many of the principles of Resharper have been added to the professional version of PyCharm .

How To Install PyCharm

I have written a guide showing how to get PyCharm, download it, extract the files and run it.

Simply click this link .

The Welcome Screen

When you first run PyCharm or when you close a project you will be presented with a screen showing a list of recent projects.

You will also see the following menu options:

There is also a configure settings option which lets you set up the default Python version and other such settings.

Creating A New Project

When you choose to create a new project you are provided with a list of possible project types as follows:

This isn't a programming tutorial so I won't be listing what all of those project types are. If you want to create a base desktop application which will run on Windows, Linux and Mac then you can choose a Pure Python project and use QT libraries to develop graphical applications which look native to the operating system they are running on regardless as to where they were developed.

As well as choosing the project type you can also enter the name for your project and also choose the version of Python to develop against.

Open A Project

You can open a project by clicking on the name within the recently opened projects list or you can click the open button and navigate to the folder where the project you wish to open is located.

Checking Out From Source Control

PyCharm provides the option to check out project code from various online resources including GitHub , CVS , Git, Mercurial and Subversion .

The PyCharm IDE

The PyCharm IDE starts with a menu at the top and underneath this you have tabs for each open project.

On the right side of the screen are debugging options for stepping through code.

The left pane has a list of project files and external libraries.

To add a file you right-click on the project name and choose "new". You then get the option to add one of the following file types:

When you add a file, such as a python file you can start typing into the editor in the right panel.

The text is all colour coded and has bold text . A vertical line shows the indentation so you can be sure that you are tabbing correctly.

The editor also includes full intellisense which means as you start typing the names of libraries or recognised commands you can complete the commands by pressing tab.

Debugging The Application

You can debug your application at any point by using the debugging options in the top right corner.

If you are developing a graphical application then you can simply press the green button to run the application. You can also press shift and F10.

To debug the application you can either click the button next to the green arrow or press shift and F9.You can place breakpoints in the code so that the program stops on a given line by clicking in the grey margin on the line you wish to break at.

To make a single step forward you can press F8 which steps over the code. This means it will run the code but it won't step into a function. To step into the function you would press F7. If you are in a function and want to step out to the calling function press shift and F8.

At the bottom of the screen whilst you are debugging you will see various windows such as a list of processes and threads, and variables that you are watching the values for.

As you are stepping through code you can add a watch on a variable so that you can see when the value changes.

Another great option is to run the code with coverage checker. The programming world has changed a lot during the years and now it is common for developers to perform test-driven development so that every change they make they can check to make sure they haven't broken another part of the system.

The coverage checker actually helps you to run the program, perform some tests and then when you have finished it will tell you how much of the code was covered as a percentage during your test run.

There is also a tool for showing the name of a method or class, how many times the items were called, and how long was spent in that particular piece of code.

Code Refactoring

A really powerful feature of PyCharm is the code refactoring option.

When you start to develop code little marks will appear in the right margin. If you type something which is likely to cause an error or just isn't written well then PyCharm will place a coloured marker.

Clicking on the coloured marker will tell you the issue and will offer a solution.

For example, if you have an import statement which imports a library and then don't use anything from that library not only will the code turn grey the marker will state that the library is unused.

Other errors that will appear are for good coding such as only having one blank line between an import statement and the start of a function. You will also be told when you have created a function that isn't in lowercase.

You don't have to abide by all of the PyCharm rules. Many of them are just good coding guidelines and are nothing to do with whether the code will run or not.

The code menu has other refactoring options. For example,​ you can perform code cleanup and you can inspect a file or project for issues.

Summary

PyCharm is a great editor for developing Python code in Linux and there are two versions available. The community version is for the casual developer whereas the professional environment provides all the tools a developer could need for creating professional software.

[Dec 12, 2015] 11 New Open Source Development Tools

Datamation

YAPF

Short for "Yet Another Python Formatter," YAPF reformats Python code so that it conforms to the style guide and looks good. It's a Google-owned project. Operating System: OS Independent

Python Execute Unix - Linux Command Examples

The os.system has many problems and subprocess is a much better way to executing unix command. The syntax is:
import subprocess
subprocess.call("command1")
subprocess.call(["command1", "arg1", "arg2"]) 

In this example, execute the date command:

 
import subprocess
subprocess.call("date")
 

Sample outputs:

Sat Nov 10 00:59:42 IST 2012
0

You can pass the argument using the following syntax i.e run ls -l /etc/resolv.conf command:

 
import subprocess
subprocess.call(["ls", "-l", "/etc/resolv.conf"])
 

Sample outputs:

<-rw-r--r-- 1 root root 157 Nov  7 15:06 /etc/resolv.conf
0

[Aug 20, 2015] Fluent Python

www.amazon.com

Paperback: 770 pages
Publisher: O'Reilly Media; 1 edition (August 20, 2015)
Language: English
ISBN-10: 1491946008
ISBN-13: 978-1491946008
Product Dimensions: 9.2 x 7 x 1.4 inches

Jascha Casadio on October 30, 2015

An excellent text covering very advanced Python features.

Among the books that are currently populating my Goodread's wishlist are no less than 20 titles dedicated to the Python language. They range from Django up to pandas, passing through Twisted and Test-Driver Development. Time is limited, so they often end up waiting in queue for months. But when I've seen Fluent Python on that shelf I had to make it mine immediately and put it in front of that queue. Getting through this book took me several months, not only because we are talking about some 700 hundreds good pages, but mostly due to the fact that it covers advanced topics that most of the Pythonists currently living on planet Earth never heard of in their life. Fluent Python is one of those books that you must taste little by little or you get devoured by those fierce topics and examples.

Released late this summer, Fluent Python is the latest work of Ramalho, a name that should sound familiar to those that have been already diving deeply into, allow me the term, Python's high-end features, powerful things, such as coroutines, that most developers never heard of in their life. Those that did probably hope never being tested on them during a job interview. And that's pretty much what the book is all about. Neither style nor the the basics of the language, but very advanced features. Quite a rare book indeed, since almost all of the Python books available introduce the readers to the language and don't get past Object Oriented Programming.

An excellent text overall, no doubts. Not for the faint of heart. Still, I am a bit puzzled by the fact that some chapters look extremely simple, others cover quirks and intricacies that you can probably live without, unless you dare touching the very core of the language ,and that get you to reach the end of a chapter with that what the hell expression on your face. The chapter covering abstract classes is an example of the former. Don't get me wrong, it's interesting and the examples well laid out. Still, it looks a basic concept that doesn't fit this kind of book.

A couple of words on the examples: they are throughout the whole book well done. The author often presents the same concepts in different flavors or does work on the same example and improves it as concepts are taken into the discussion. The code is intense but easy to follow. Key lines are extensively explained later on, so that the reader won't miss that specific features that makes it all possible. There are so many gems that you will probably end up writing most of that code down to make it yours. This is actually the best thing the reader can do. Try it, modify it, assimilate it, master it.

Among the many topics covered there are two that are worth mentioning: the first is chapter four, which covers strings, Unicode and bytes. Marvelous, simply marvelous. The examples, the explanations. So clear and to the point. You definitely get away from it with a deep understanding of how strings work in Python 2.7 and 3.

The second is that dedicated to futures. Actually it's the whole topic, which spans several chapters at the very end of the book. The authors shows how working with threads and subprocesses improve the efficiency of an application, and how easy it is to exploit them through the futures that are now available in the language. He does gives us a very interesting example in many different flavors, showing us how the code and performance change. Great.

Decorators and closures are also well described, even if not as good as the aforementioned topics. In that sense, the author does complement what we find about the subject in Effective Python: 59 Specific Ways to Write Better Python, another must have for any serious Pythonist.

Overall, a great Python book. A must have for any Python developer interested in getting the most out of the language.

As usual, you can find more reviews on my personal blog: http://books.lostinmalloc.com . Feel free to pass by and share your thoughts!

[Dec 04, 2014] Introducing Python Modern Computing in Simple Packages by Bill Lubanovic

This is a good intro book that covered Python 3 ! An excellent book for beginners.The opening chapters of this book provide a very good overview of Python syntax, methods and structures, delivered with wit. Examples range from simple to quite complex.
Notable quotes:
"... The author didn't do a good job in wrapping the teaching around practical examples for a huge portion of the book, but rather, resorted to the lazy approach many coding-book authors take: plop some super-synthetic code on the page and explain it. They have no connection to reality. ..."
www.amazon.com

Paperback: 478 pages
Publisher: O'Reilly Media; 1 edition (December 4, 2014)
Language: English
ISBN-10: 1449359361
ISBN-13: 978-1449359362
Product Dimensions: 7 x 1 x 9.2 inches

About the Author

Bill Lubanovic has developed software with UNIX since 1977, GUIs since 1981, databases since 1990, and the Web since 1993.

At a startup named Intran in 1982, he developed MetaForm -- one of the first commercial GUIs (before the Mac or Windows), on one of the first graphic workstations. At Northwest Airlines in the early 1990s, he wrote a graphic yield management system that generated millions of dollars in revenue; got the company on the Internet; and wrote its first Internet marketing test. He co-founded an ISP (Tela) in 1994, and a web development company (Mad Scheme) in 1999.

Recently, he developed core services and distributed systems with a remote team for a Manhattan startup. Currently, he's integrating OpenStack services for a supercomputer company.

He enjoys life in Minnesota with his wonderful wife Mary, children Tom and Karin, and cats Inga, Chester, and Lucy.

ByTS.on August 23, 2015

another book with useless synthetic examples

The little code snippets that the author provides are useful to understand the commands, but without putting it in the context of some useful practical code, it makes the book very dry. Although I understood most of the concepts from the snippets, I was dragging myself through the book since it was so boring to read.

The author didn't do a good job in wrapping the teaching around practical examples for a huge portion of the book, but rather, resorted to the lazy approach many coding-book authors take: plop some super-synthetic code on the page and explain it. They have no connection to reality.

The only reason I am not giving a single star is because the book was cheap and I learned "some" stuff. But these are all the wrong reasons to like a book for !

Alfredzo Nash on March 4, 2015

An Edible Introduction to Python

Reviewed by Alfredzo Nash, Fairfield County Area Datto Linux User Group.

"Introducing Python" by Bill Lubanovic is an edible recipe for learning Python. I had limited exposure to Python (I completed 50% of the Codecademy training) prior to reading "Introducing Python". Lubanovic is ab iron chef that has written this delectable book.

Beginning with Chapters 1-5 (1. A Taste of Python, 2. Python Ingredients, 3. PyFilling, 4. PyCrust, and 5. PyBoxes), the reader starts to build their mental palate using a small cookbook style dialog for making an actual pie! This was very informative and one of the best analogies within the book. Lubanovic skillfully separates the importance of data structure such as list, dictionaries, tuples, and set from code structure (commenting, if elif else, utf-8 encoding, pep, etc.) which allowed me to become familiar with the essential building blocks of the Python language. "Introducing Python" is a very definitive guide to what makes Python a valuable and powerful language.

Since Python 2.7's end-of-life is scheduled in 2020, Lubanovic encouraged Pythonistas to begin writing Python code using Python 3 instead of 2.7. Being relatively new to the language, I couldn't resist the urge to discover more about the differences between the two versions. At the time of this writing, there are still subtle differences between Python 2.7 and Python 3. The changes include how to call the print() function, and the handling of Unicode characters. Most of the differences between version 2.7 and 3.0 can be found at https://wiki.python.org/moin/Python2orPython3.

Unfortunately, Chapters 6-9 (6. Oh Oh: Objects and Classes, 7. Mangle Data Like a Pro, 8. Data Has to Go Somewhere, 9. The Web Untangled) proved difficult for me to comprehend due to my inexperience. Lubanovic attempts to ease the reader into this section skillfully in Chapter 6, with analogies such as "In Chapter 1, I compare an object to a plastic box. A class is like the mold that makes that box." I couldn't agree with this analogy as I had to learn the complex object-oriented terminology, like polymorphism, instantiation and inheritance, discussed throughout this chapter.

Chapters 7-8 puts "Introducing Python" into high gear. I did not take the term "Pro" lightly within this section. Essentially, I now understand why data scientist, forensic analysts, cloud engineers, and automation engineers consider Python for their projects. Data can take many forms that include text strings, ASCII or binary. These types of data need to be written, read, encoded, decoded and stored properly. Lubanovic clearly conveyed the message that data should be handled with precision. UTF-8 encoding, byte arrays, and matching regular expression all took some time for me to understand. There are tons of options that include storing data between RAM, CSVs, JSON, or SQL and NoSQL Databases. Both mangling and storing data requires a certain level of mastery that has convinced me to keep Introducing Python as a reminder to sharpen my skills in this area of expertise.

Chapters 9-12 (9.The Web, Untangled, 10. Systems, 11. Concurrency and Networks,12. Be a Pythonista), are my favorite chapters in Introducing Python. Chapter 9 explains how Python's standard Web libraries handle the various components of the Web. It describes mainly the http and urlib packages, but also makes notable references to web frameworks such as Bottle and Flask. Chapter 10 dived into how Python handles files, directories, processes and time using the os (Operating System) module. The os module contains the functions copy(),chown(), remove(), mkdir(),rmdir() and many more that operate the same way as their Linux/UNIX counterparts.

Chapter 11 reinforces Chapter 10's section on process management via the os module and using multiprocessing. Diving deeper into the concurrency standard library (http://bit.ly/concur-lib) within Python, Lubanovic discusses the differences between queues, process, and threads, as well as the tools to manage them on a single machine or multiple machines. Noticabley, Redis, gevent, asyncio, and twisted are mentioned as additional tools that handle concurrency within Python as well. In the Networks section, Lubanovic explores the plumbing provided by the numerous Internet Services supported within Python. TCP/IP, Sockets, Web APIs (fabric), RPCs and Message Brokers (RabbitMQ /ZeroMQ) are all covered in great detail. This chapter is an invaluable resource for anyone interested in gaining a more in-depth knowledge of how use Python to manage concurrency across distributed systems and networks.

Chapter 12 sets the example for ALL Python users, by encouraging them to "Be A Pythonista". Noticeably, Being a Pythonista is no small task. The requirements consist of: Finding Code, Installing Packages, Documenting Your Code, Testing Your code, Optimizing your code, and Managing your code via Source Control Whew... Humbly, I accept this challenge and hope to become a proficient Pythonista and encourage everyone interested in Python to do the same.

In closing, Introducing Python was remarkable read for me. I recommend studying this book and holding onto for future reference in real-world scenarios.

[Feb 09, 2014] build - How to package Python as RPM for install into -opt

Stack Overflow

Q: How to create a binary RPM package out of Python 2.7.2 sources for installation into a non-standard prefix such as /opt/python27?

Assume the following builds correctly.

wget http://python.org/ftp/python/2.7.2/Python-2.7.2.tgz
tar zxvf Python-2.7.2.tgz
cd Python-2.7.2
./configure --prefix=/opt/python27 --enable-shared
make
make test
sudo make install

Instead of the last command I'd like to build a binary RPM.

A:

RPMs are built using rpmbuild from a .spec file. As an example, look at python.spec from Fedora.

If you don't need to build from sources then try rpm's --relocate switch on a pre-built RPM for your distribution:

rpm -i --relocate /usr=/opt/python27 python-2.7.rpm

Python 2.7 RPMs - Gitorious

Python 2.7 RPMs

Port of the Fedora 15 Python 2.7 RPM and some of the related stack to build on RHEL 5 & 6 (and derivatives such as CentOS). Can be installed in parallel to the system Python packages.

[Feb 09, 2014] Building and Installing Python 2.7 RPMs on CentOS 5.7 by Nathan Milford

blog.milford.io
I was asked today to install Python 2.7 on a CentOS based node and I thought I'd take this oportunity to add a companion article to my Python 2.6 article.

We're all well aware that CentOS is pretty backwards when it comes to having the latest and greatest sotware packages and is particularly finicky when it comes to Python since so much of RHEL depends on it.

As a rule, I refuse to rush in and install anything in production that isn't in a manageable package format such as RPM. I need to be able to predictably reproduce software installs across a large number of nodes.

The following steps will not clobber your default Python 2.4 install and will keep both CentOS and your developers happy.

So, here we go.

Install the dependancies.

sudo yum -y install rpmdevtools tk-devel tcl-devel expat-devel db4-devel \
                    gdbm-devel sqlite-devel bzip2-devel openssl-devel \
                    ncurses-devel readline-devel

Setup you RPM build envirnoment.

rpmdev-setuptree

Grab my spec file.

wget https://raw.github.com/nmilford/specfiles/master/python-2.7/python27-2.7.2.spec \
     -O ~/rpmbuild/SPECS/python27-2.7.2.spec 
wget http://www.python.org/ftp/python/2.7.2/Python-2.7.2.tar.bz2 \
     -O ~/rpmbuild/SOURCES/Python-2.7.2.tar.bz2

Build RPM. (FYI, the QA_RPATHS variable tells the rpmbuild to skip some file path errors).

QA_RPATHS=$[ 0x0001|0x0010 ] rpmbuild -bb ~/rpmbuild/SPECS/python-2.7.2.spec

Install the RPMs.

sudo rpm -Uvh ~/rpmbuild/RPMS/x86_64/python27*.rpm

Now on to the the setuptools.

Grab my spec file.

wget https://raw.github.com/nmilford/specfiles/master/python-2.7/python27-setuptools-0.6c11.spec \
     -O ~/rpmbuild/SPECS/python27-setuptools-0.6c11.spec 

Grab the source.

wget http://pypi.python.org/packages/source/s/setuptools/setuptools-0.6c11.tar.gz \
     -O ~/rpmbuild/SOURCES/setuptools-0.6c11.tar.gz

Build the RPMs.

rpmbuild -bb ~/rpmbuild/SPECS/python27-setuptools-0.6c11.spec

Install the RPMs.

sudo rpm -Uvh ~/rpmbuild/RPMS/noarch/python27-setuptools-0.6c11-milford.noarch.rpm

Now, we'll install MySQL-python as an example.

Grab the mysql-dev package

yum -y install mysql-devel

Grab, build and install the MySQL-python package.

curl http://superb-sea2.dl.sourceforge.net/project/mysql-python/mysql-python/1.2.3/MySQL-python-1.2.3.tar.gz | tar zxv
cd MySQL-python-1.2.3
python2.7 setup.py build
python2.7 setup.py install

Like with the previous Python 2.6 article, note how I called the script explicitly using the following python binary: /usr/bin/python2.7

Now we're good to give it the old test thus:

python2.7 -c "import MySQLdb"

If it doesn't puke out some error message, you're all set.

Happy pythoning.

List of Python software - Wikipedia, the free encyclopedia

[Nov 01, 2012] Python for Data Analysis

www.amazon.com

Paperback: 466 pages
Publisher: O'Reilly Media; 1 edition (November 1, 2012)
Language: English
ISBN-10: 1449319793
ISBN-13: 978-1449319793
Product Dimensions: 7 x 0.9 x 9.2 inches

R. Friesel Jr. on October 22, 2012

dive into pandas and NumPy

Wes McKinney's "Python for Data Analysis" (O'Reilly, 2012) is a tour pandas and NumPy (mostly pandas) for folks looking to crunch "big-ish" data with Python. The target audience is not Pythonistas, but rather scientists, educators, statisticians, financial analysts, and the rest of the "non-programmer" cohort that is finding more and more these days that it needs to do a little bit-sifting to get the rest of their jobs done.

First, two warnings:

1. **This book is not an introduction to Python.** While McKinney does not assume that you know *any* Python, he isn't exactly going to hold your hand on the language here. There is an appendix ("Python Language Essentials") that beginners will want to read before getting too far, but otherwise you're on your own. ("Lucky for you Python is executable pseudocode"?)

2. **This book is not about theories of data analysis.** What I mean by that is: if you're looking for a book that is going to tell you the *types* of analyses to do, this is not that book. McKinney assumes that you already know, through your "actual" training, what kinds of analyses you need to perform on your data, and how to go about the computations necessary for those analyses.

That being said: McKinney is the principal author on pandas, a Python package for doing data transformation and statistical analysis. The book is largely about pandas (and NumPy), offering overviews of the utilities in these packages, and concrete examples on how to employ them to great effect. In examining these libraries, McKinney also delves into general methodologies for munging data and performing analytical operations on them (e.g., normalizing messy data and turning it into graphs and tables). McKinney also delves into some (semi) esoteric information about how Python works at very low levels and ways to optimize data structures so that you can get maximum performance from your programs. McKinney is clearly knowledgeable about these libraries, about Python, and about using those tools effectively in analytical software.

So where do I land on "Python for Data Analysis"? If you're looking for a book that discusses data analysis in a broad sense, or one that pays special attention to the theory, this isn't that book. If you're looking for a generalist's book on Python--also not this book. However, if you've already selected Python as your analytical tool (and it sounds like it's more/less the de facto analytical tool in many circles) then this just might be the perfect book for you.

[Oct 21, 2012] Last File Manager freshmeat.net

Written in Python. The last version is LFM 2.3 dated May 2011. Codebase is below 10K lines.
Lfm is a curses-based file manager for the Unix console written in Python

21 May 2011

Python 2.5 or later is required now. PowerCLI was added, an advanced command line interface with completion, persistent history, variable substitution, and many other useful features.

Persistent history in all forms was added. Lots of improvements were made and bugs were fixed

[Aug 23, 2012] Think Python by Allen B. Downey

Notable quotes:
"... "Think Python" is available online ([...]) which means you can decide if you like it first. ..."
"... most importantly, it is NOT a "Learn Python in X days" type book. Those have their place, but this book targets those who actually are/want to be developers. Hence the subtitle "How to Think Like a Computer Scientist." ..."
"... Each chapter ends with debugging tips ..."
"... I think this makes for a great first Python book. To be followed by one that teaches the Python libraries. It teaches you how to think in Python. And how to be a developer; not just a coder. ..."
www.amazon.com

Paperback: 300 pages
Publisher: O'Reilly Media; 1 edition (August 23, 2012)
Language: English
ISBN-10: 144933072X
ISBN-13: 978-1449330729
Product Dimensions: 7 x 0.9 x 9.2 inches

Jeanne Boyarsky on September 8, 2012

development vs coding

"Think Python" is available online ([...]) which means you can decide if you like it first. Personally, I wanted to write in my copy making the paper copy a great thing. Inexpensive too for a computer book. It's one of those great books I know I'll refer to again. Can't imagine why you'd buy the Kindle version though.

The book is targetted at those learning Python. It's appropriate whether you are new to programming or coming from another language. And most importantly, it is NOT a "Learn Python in X days" type book. Those have their place, but this book targets those who actually are/want to be developers. Hence the subtitle "How to Think Like a Computer Scientist."

Each chapter ends with debugging tips, a glossary of terms and numerous exercises for practice. Common idioms are covered in addition to syntax, techniques and algorithms. Recursion is presented in a not scary, approachable way.

The author uses the term "state diagram" to refer to the state of variables in an object. I've never seen this usage before (being more used to the UML state diagram) and look forward to asking the author about it in his coderanch.com book promotion next month.

I think this makes for a great first Python book. To be followed by one that teaches the Python libraries. It teaches you how to think in Python. And how to be a developer; not just a coder.

Disclosure: I received a copy of this book from the publisher in exchange for writing this review.

[Oct 06, 2011] Text Processing in Python (a book)

stackoverflow.com

A couple of you make donations each month (out of about a thousand of you reading the text each week). Tragedy of the commons and all that... but if some more of you would donate a few bucks, that would be great support of the author.

In a community spirit (and with permission of my publisher), I am making my book available to the Python community. Minor corrections can be made to later printings, and at the least errata noted on this website. Email me at <mertz@gnosis.cx> .

A few caveats:

(1) This stuff is copyrighted by AW (except the code samples which are released to the public domain). Feel free to use this material personally; but no permission is given for further distribution beyond your personal use.

(2) The book is provided in "smart ASCII" format. This is converted to print (and maybe to fancier electronic formats) by automated scripts (txt->LaTeX->PDF for the printed version).

As a highly sophisticated "digital rights management" system, those scripts are not themselves made readily available. :-)

acknowledgments.txt FOLKS WHO HAVE MADE THIS BOOK BETTER
intro.txt INTRODUCTION
chap1.txt PYTHON BASICS
chap2.txt BASIC STRING OPERATIONS
chap3.txt REGULAR EXPRESSIONS
chap4.txt PARSERS AND STATE-MACHINES
chap5.txt INTERNET TOOLS AND TECHNIQUES
appendix_a.txt A SELECTIVE AND IMPRESSIONISTIC SHORT REVIEW OF PYTHON
appendix_b.txt A DATA COMPRESSION PRIMER
appendix_c.txt UNDERSTANDING UNICODE
appendix_d.txt A STATE-MACHINE FOR ADDING MARKUP TO TEXT
glossary.txt GLOSSARY TERMS

[Oct 06, 2011] Text Processing in Python (a book)

A couple of you make donations each month (out of about a thousand of you reading the text each week). Tragedy of the commons and all that... but if some more of you would donate a few bucks, that would be great support of the author.

In a community spirit (and with permission of my publisher), I am making my book available to the Python community. Minor corrections can be made to later printings, and at the least errata noted on this website. Email me at <mertz@gnosis.cx> .

A few caveats:

(1) This stuff is copyrighted by AW (except the code samples which are released to the public domain). Feel free to use this material personally; but no permission is given for further distribution beyond your personal use.

(2) The book is provided in "smart ASCII" format. This is converted to print (and maybe to fancier electronic formats) by automated scripts (txt->LaTeX->PDF for the printed version).

As a highly sophisticated "digital rights management" system, those scripts are not themselves made readily available. :-)

acknowledgments.txt FOLKS WHO HAVE MADE THIS BOOK BETTER
intro.txt INTRODUCTION
chap1.txt PYTHON BASICS
chap2.txt BASIC STRING OPERATIONS
chap3.txt REGULAR EXPRESSIONS
chap4.txt PARSERS AND STATE-MACHINES
chap5.txt INTERNET TOOLS AND TECHNIQUES
appendix_a.txt A SELECTIVE AND IMPRESSIONISTIC SHORT REVIEW OF PYTHON
appendix_b.txt A DATA COMPRESSION PRIMER
appendix_c.txt UNDERSTANDING UNICODE
appendix_d.txt A STATE-MACHINE FOR ADDING MARKUP TO TEXT
glossary.txt GLOSSARY TERMS

[Apr 04, 2011] Scripting the Linux desktop, Part 1 Basics by Paul Ferrill

Jan 18, 2011 | developerWorks

Developing applications for the Linux desktop typically requires some type of graphical user interface (GUI) framework to build on. Options include GTK+ for the GNOME desktop and Qt for the K Desktop Environment (KDE). Both platforms offer everything a developer needs to build a GUI application, including libraries and layout tools to create the windows users see. This article shows you how to build desktop productivity applications based on the screenlets widget toolkit (see Resources for a link).

A number of existing applications would fit in the desktop productivity category, including GNOME Do and Tomboy. These applications typically allow users to interact with them directly from the desktop through either a special key combination or by dragging and dropping from another application such as Mozilla Firefox. Tomboy functions as a desktop note-taking tool that supports dropping text from other windows.

Getting started with screenlets

You need to install a few things to get started developing screenlets. First, install the screenlets package using either the Ubuntu Software Center or the command line. In the Ubuntu Software Center, type screenlets in the Search box. You should see two options for the main package and a separate installation for the documentation.

Python and Ubuntu

You program screenlets using Python. The basic installation of Ubuntu 10.04 has Python version 2.6 installed, as many utilities depend on it. You may need additional libraries depending on your application's requirements. For the purpose of this article, I installed and tested everything on Ubuntu version 10.04.

Next, download the test screenlet's source from the screenlets.org site. The test screenlet resides in the src/share/screenlets/Test folder and uses Cairo and GTK, which you also need to install. The entire source code for the test program is in the TestScreenlet.py file. Open this file in your favorite editor to see the basic structure of a screenlet.

Python is highly object oriented and as such uses the class keyword to define an object. In this example, the class is named TestScreenlet and has a number of methods defined. In TestScreenlet.py, note the following code at line 42:

def __init__(self, **keyword_args):
Python uses the leading and trailing double underscore (__) notation to identify system functions with predefined behaviors. In this case, the __init__ function is for all intents and purposes the constructor for the class and contains any number of initialization steps to be executed on the creation of a new instance of the object. By convention, the first argument of every class method is a reference to the current instance of the class and is named self. This behavior makes it easy to use self to reference methods and properties of the instance it is in:
self.theme_name = "default"
The screenlets framework defines several naming conventions and standards, as outlined on screenlets.org's developer's page (see Resources for a link). There's a link to the source code for the screenlets package along with the application programming interface (API) documentation. Looking at the code also gives you insight into what each function does with the calling arguments and what it returns.

Writing a simple screenlet

The basic components of a screenlet include an icon file, the source code file, and a themes folder. The themes folder contains additional folders for different themes. You'll find a sample template at screenlets.org with the required files and folders to help you get started.

For this first example, use the template provided to create a basic "Hello World" application. The code for this basic application is shown in Listing 1.

Listing 1. Python code for the Hello World screenlet
	
#!/usr/bin/env python

import screenlets

class HelloWorldScreenlet(screenlets.Screenlet):
    __name__ = 'HelloWorld'
    __version__ = '0.1'
    __author__ = 'John Doe'
    __desc__ = 'Simple Hello World Screenlet'
    
    def __init__(self, **kwargs):
        # Customize the width and height.
        screenlets.Screenlet.__init__(self, width=180, height=50, **kwargs)
    
    def on_draw(self, ctx):
        # Change the color to white and fill the screenlet.
        ctx.set_source_rgb(255, 255, 255)
        self.draw_rectangle(ctx, 0, 0, self.width, self.height)

        # Change the color to black and write the message.
        ctx.set_source_rgb(0, 0, 0)
        text = 'Hello World!'
        self.draw_text(ctx, text, 10, 10, "Sans 9" , 20, self.width)


if __name__ == "__main__":
    import screenlets.session
    screenlets.session.create_session(HelloWorldScreenlet)

Each application must import the screenlets framework and create a new session. There are a few other minimal requirements, including any initialization steps along with a basic draw function to present the widget on screen. The TestScreenlet.py example has an __init__ method that initializes the object. In this case, you see a single line with a call to the screenlet's __init__ method, which sets the initial width and height of the window to be created for this application.

The only other function you need for this application is the on_draw method. This routine sets the background color of the box to white and draws a rectangle with the dimensions defined earlier. It sets the text color to black and the source text to "Hello World!" and then draws the text. ...

Reusing code in a more complex screenlet

One nice thing about writing screenlets is the ability to reuse code from other applications. Code reuse opens a world of possibilities with the wide range of open source projects based on the Python language. Every screenlet has the same basic structure but with more methods defined to handle different behaviors. Listing 2 shows a sample application named TimeTrackerScreenlet.

Listing 2. Python code for the Time Tracker screenlet

	
#!/usr/bin/env python

import screenlets
import cairo
import datetime

class TimeTrackerScreenlet(screenlets.Screenlet):
	__name__ = 'TimeTrackerScreenlet'
	__version__ = '0.1'
	__author__ = 'John Doe'
	__desc__ = 'A basic time tracker screenlet.'
	
	theme_dir = 'themes/default'
	image = 'start.png'

	def __init__(self, **keyword_args):
		screenlets.Screenlet.__init__(self, width=250, height=50, **keyword_args)
		self.add_default_menuitems()
		self.y = 25
		self.theme_name = 'default'
		self.on = False
		self.started = None

	def on_draw(self, ctx):
		self.draw_scaled_image(ctx, 0, 0, self.theme_dir + '/' + 
		self.image, self.width, self.height)
		
	def on_mouse_down(self, event):
		if self.on:
			self.started = datetime.datetime.now()
			self.image = 'stop.png'
			self.on = False
		else:
			if self.started:
				length = datetime.datetime.now() - self.started
				screenlets.show_message(None, '%s seconds' % 
				length.seconds, 'Time')
				self.started = None
			self.image = 'start.png'
			self.on = True

	def on_draw_shape(self, ctx):
		self.on_draw(ctx)
		ctx.rectangle(0, 0, self.width, self.height)
		ctx.fill()
	

if __name__ == "__main__":
	import screenlets.session
	screenlets.session.create_session(TimeTrackerScreenlet)
This example introduces a few more concepts that you need to understand before you start building anything useful. All screenlet applications have the ability to respond to specific user actions or events such as mouse clicks or drag-and-drop operations. In this example, the mouse down event is used as a trigger to change the state of your icon. When the screenlet runs, the start.png image is displayed. Clicking the image changes it to stop.png and records the time started in self.started. Clicking the stop image changes the image back to start.png and displays the amount of time elapsed since the first start image was clicked.

Responding to events is another key capability that makes it possible to build any number of different applications. Although this example only uses the mouse_down event, you can use the same approach for other events generated either by the screenlets framework or by a system event such as a timer. The second concept introduced here is persistent state. Because your application is running continuously, waiting for an event to trigger some action, it is able to keep track of items in memory, such as the time the start image was clicked. You could also save information to disk for later retrieval, if necessary.

Automating tasks with screenlets

Now that you have the general idea behind developing screenlets, let's put all together. Most users these days use a Really Simple Syndication (RSS) reader to read blogs and news feeds. For this last example, you're going to build a configurable screenlet that monitors specific feeds for keywords and displays any hits in a text box. The results will be clickable links to open the post in your default Web browser. Listing 3 shows the source code for the RSS Search screenlet.

Listing 3. Python code for the RSS Search screenlet
	
#!/usr/bin/env python

from screenlets.options import StringOption, IntOption, ListOption
import xml.dom.minidom
import webbrowser
import screenlets
import urllib2
import gobject
import pango
import cairo

class RSSSearchScreenlet(screenlets.Screenlet):
    __name__ = 'RSSSearch'
    __version__ = '0.1'
    __author__ = 'John Doe'
    __desc__ = 'An RSS search screenlet.'
    
    topic = 'Windows Phone 7'
    feeds = ['http://www.engadget.com/rss.xml',
             'http://feeds.gawker.com/gizmodo/full']
    interval = 10
    
    __items = []
    __mousesel = 0
    __selected = None
    
    def __init__(self, **kwargs):
        # Customize the width and height.
        screenlets.Screenlet.__init__(self, width=250, height=300, **kwargs)
        self.y = 25
        
    def on_init(self):
        # Add options.
        self.add_options_group('Search Options',
                               'RSS feeds to search and topic to search for.')
        self.add_option(StringOption('Search Options',
            'topic',
            self.topic,
            'Topic',
            'Topic to search feeds for.'))
        self.add_option(ListOption('Search Options',
                                   'feeds',
                                   self.feeds,
                                   'RSS Feeds',
                                   'A list of feeds to search for a topic.'))
        self.add_option(IntOption('Search Options',
                                  'interval',
                                  self.interval,
                                  'Update Interval',
                                  'How frequently to update (in seconds)'))

        self.update()
        
    def update(self):
        """Search selected feeds and update results."""
        
        self.__items = []

        # Go through each feed.
        for feed_url in self.feeds:
            
            # Load the raw feed and find all item elements.
            raw = urllib2.urlopen(feed_url).read()
            dom = xml.dom.minidom.parseString(raw)
            items = dom.getElementsByTagName('item')
            
            for item in items:
                
                # Find the title and make sure it matches the topic.
                title = item.getElementsByTagName('title')[0].firstChild.data
                if self.topic.lower() not in title.lower(): continue
                
                # Shorten the title to 30 characters.
                if len(title) > 30: title = title[:27]+'...'
                
                # Find the link and save the item.
                link = item.getElementsByTagName('link')[0].firstChild.data
                self.__items.append((title, link))

        self.redraw_canvas()

        # Set to update again after self.interval.
        self.__timeout = gobject.timeout_add(self.interval * 1000, self.update)
        
    def on_draw(self, ctx):
        """Called every time the screenlet is drawn to the screen."""
        
        # Draw the background (a gradient).
        gradient = cairo.LinearGradient(0, self.height * 2, 0, 0)
        gradient.add_color_stop_rgba(1, 1, 1, 1, 1)
        gradient.add_color_stop_rgba(0.7, 1, 1, 1, 0.75)
        ctx.set_source(gradient)
        self.draw_rectangle_advanced (ctx, 0, 0, self.width - 20,
                                      self.height - 20,
                                      rounded_angles=(5, 5, 5, 5),
                                      fill=True, border_size=1,
                                      border_color=(0, 0, 0, 0.25),
                                      shadow_size=10,
                                      shadow_color=(0, 0, 0, 0.25))
        
        # Make sure we have a pango layout initialized and updated.
        if self.p_layout == None :
            self.p_layout = ctx.create_layout()
        else:
            ctx.update_layout(self.p_layout)
            
        # Configure fonts.
        p_fdesc = pango.FontDescription()
        p_fdesc.set_family("Free Sans")
        p_fdesc.set_size(10 * pango.SCALE)
        self.p_layout.set_font_description(p_fdesc)

        # Display our text.
        pos = [20, 20]
        ctx.set_source_rgb(0, 0, 0)
        x = 0
        self.__selected = None
        for item in self.__items:
            ctx.save()
            ctx.translate(*pos)
            
            # Find if the current item is under the mouse.
            if self.__mousesel == x and self.mouse_is_over:
                ctx.set_source_rgb(0, 0, 0.5)
                self.__selected = item[1]
            else:
                ctx.set_source_rgb(0, 0, 0)
            
            self.p_layout.set_markup('%s' % item[0])
            ctx.show_layout(self.p_layout)
            pos[1] += 20
            ctx.restore()
            x += 1

    def on_draw_shape(self, ctx):
        ctx.rectangle(0, 0, self.width, self.height)
        ctx.fill()
        
    def on_mouse_move(self, event):
        """Called whenever the mouse moves over the screenlet."""
        
        x = event.x / self.scale
        y = event.y / self.scale
        self.__mousesel = int((y -10 )/ (20)) -1
        self.redraw_canvas()
    
    def on_mouse_down(self, event):
        """Called when the mouse is clicked."""
        
        if self.__selected and self.mouse_is_over:
            webbrowser.open_new(self.__selected)


if __name__ == "__main__":
    import screenlets.session
    screenlets.session.create_session(RSSSearchScreenlet)

Building on the concepts of the first two examples, this screenlet uses a number of new concepts, including the config page. In the on_init routine, three options are added for the user to specify: a list of RSS feeds to track, a topic of interest to search for, and an update interval. The update routine then uses all of these when it runs.

Python is a great language for this type of task. The standard library includes everything you need to load the Extensible Markup Language (XML) from an RSS feed into a searchable list. In Python, this takes just three lines of code:

raw = urllib2.urlopen(feed_url).read()
dom = xml.dom.minidom.parseString(raw)
items = dom.getElementsByTagName('item')
The libraries used in these three lines include urllib2 and xml. In the first line, the entire contents found at the feed_url address are read into the string raw. Next, because you know that this string contains XML, you use the Python XML library dom.minidom.parseString method to create a document object made up of node objects.

Finally, you create a list of element objects corresponding to the individual XML elements named item. You can then iterate over this list to search for your target topic. Python has a very elegant way of iterating over a list of items using the for keyword, as in this code snippet:

for item in items:
    # Find the title and make sure it matches the topic.
    title = item.getElementsByTagName('title')[0].firstChild.data
    if self.topic.lower() not in title.lower(): continue

Each item matching your criteria is added to the currently displayed list, which is associated with this instance of the screenlet. Using this approach makes it possible to have multiple instances of the same screenlet running, each configured to search for different topics. The final part of the update function redraws the text with the updated list and fires off a new update timer based on the interval on the config page. By default, the timer fires every 10 seconds, although you could change that to anything you want. The timer mechanism comes from the gobject library, which is a part of the GTK framework.

This application expands the on_draw method quite heavily to accommodate your new functionality. Both the Cairo and Pango libraries make it possible to create some of the effects used in the text window. Using a gradient gives the background of the widget a nice look along with rounded angles and semi-transparency. Using Pango for layout adds a number of functions for saving and restoring the current context easily. It also provides a way to generate scalable fonts based on the current size of the screenlet.

The trickiest part in the on_draw method is handling when a user hovers over an item in the list. Using the for" keyword, you iterate over the items in the screenlet to see whether the user is hovering over that particular item. If so, you set the selected property and change the color to provide visual feedback. You also use a bit of markup to set the link property to bold-probably not the most elegant or efficient way to deal with the problem, but it works. When a user clicks one of the links in the box, a Web browser is launched with the target URL. You can see this functionality in the on_mouse_down function. Python and its libraries make it possible to launch the default web browser to display the desired page with a single line of code. Figure 2 shows an example of this screenlet.

[Jan 10, 2011] Programming Python by Mark Lutz

This is a typical Mark Lutz book. Very pedantic. Not very insightful... More reference then a textbook...
Notable quotes:
"... The 2010 publication date is a paradox, because this book only covers the new Python v3, which is a major split from Python 2. ..."
"... So again, this book is a poor fit. No matter how you slice it. (rim shot) ..."
www.secnetix.de

Paperback: 1632 pages
Publisher: O'Reilly Media; Fourth Edition edition (January 10, 2011)
Language: English
ISBN-10: 0596158106
ISBN-13: 978-0596158101
Product Dimensions: 7 x 3 x 9.2 inches

All the books by this guy are written for marketing, selling, not for teaching. If you look at his website you can easily see that he is a good marketer.

Language is very bad and obscure. A lot of useless jargon. A lot of non-sense repetitions. A lot of non-sense examples. I am 100 percent sure that these stellar reviews here are not written by readers. I am very sure that these reviews are either written by the author, or by the marketing department!

Before buying this book, just read one chapter or section and you will understand what I mean.

My alternative suggestion is: Read the official tutorial on the python site. It is much better and sufficient for most purposes. And it is free. If you need a reference book, well, just type the function name or the subject into python.org.

If you are very willing to pay money, then buy a serious book, written by a well educated computer scientist, or whatever. Do not buy books written by simple-minded practitioners.

Antonio (New Zealand) - See all my reviews

3.0 out of 5 stars Reasonable source of information but some aspects I didn't like, September 21, 2011

I've programmed in Python before, but haven't used it for a couple of years. I was looking for a refresher, as well as some example applications.

Firstly note that this book isn't an introduction to Python, nor is it a reference. The author makes that clear in the preface, instead referring you to the other titles he has written. Also the book covers Python 3.x. Perhaps those who are interested in earlier versions should get the previous edition of the book. On the other hand while there are some changes between the two versions, reading the book wouldn't be a waste of time if you are interested in Python 2.x

I liked this book in the sense that if I looked up a particular topic, I often found his discussion reasonable and could get some useful idiomatic python code to use.

On the other hand, the author intends this book as a tutorial. When I tried to read through it as a tutorial I just found it falling a bit flat. Also at around 1600 pages I doubt I would have the endurance to read through it from beginning to end.

I guess the main problem with the book is that you are interested in one particular area to use Python, say web development, or interfacing with databases this book would probably have insufficient detail, and you would want a specialist book in that area. Also I found the authors writing style somewhat verbose. Another issue is that those people who want to build a GUI for instance may not be interested in his choice of tool Tkinter.

In conclusion, this book does have some useful information, I didn't really like it. While it is hard to pin down the reasons for my dislike, I guess it is because he tries to cover so many topics, that not all of them are covered that well. Also it is not always clear who the audience is, beginners may find his explanations to terse, whereas those who have some familiarity with python may wonder why he is pointing out the obvious. I recommend people who are looking to develop a particular application in python, instead get a book more focused on their area of interest. Those who are new to python should avoid this book also. Those who are looking for a python 3.x refresher should find a book that's a little less weighty.

It's kind of annoying all those people who have received a free book from O'reilly giving it a five star review. Although they disclosed it, it now makes me suspicious as to how many other five star reviews are given by people who enjoy getting free books, and haven't disclosed the fact.

Luciano Ramalho "stand-up programmer" (Sao Paulo, SP Brazil) - See all my reviews
(REAL NAME)

2.0 out of 5 stars Rambling, poorly edited, not accessible for beginners and too slow for everyone else, December 18, 2011

I own more than 100 O'Reilly books and dozens of Python books, since I've been a Python user, instructor and evangelist for 13 years now. The first edition of this book was the first book published about Python by O'Reilly, and it was often compared to Programming Perl at the time. The comparison was very bad for this book: it is much longer, yet shallower than the Camel Book; it tries hard to be funny, but Larry Wall's jokes are less frequent but more effective; it is poorly edited, while the Camel Book is a gem and a true classic.

The pace is excruciatingly slow for a seasoned programmer of any language, but in spite of long and repetitive explanations this book is not accessible to beginners because of excessive, needless jargon and attention to irrelevant details when first introducing language features, making the narrative hard to digest.

It is accurate and up to date, and for this reason I give it 2 stars instead of one. But anyone looking for a Python book will be better served looking elsewhere. From O'Reilly, Alex Martelli's Python in a Nutshell is the best there is to really understand how the language works and how it should be used, even if it is outdated. Python Essential Reference by David Beazley is excellent too, and the 4th edition is very up to date. The Dive into Python books (Python 2 and Python 3 versions exist) are also excellent, and free as in speech. The Quick Python Book by Manning is also good. In fact, every other Python book that I know is a better buy than this one, which probably sells mainly due to the O'Reilly brand and because it was the first. BTW, Martelli, on p. 12 of Python in a Nutshell, 2e, refers readers to nine other books by O'Reilly and other publishers, including two others by Mark Lutz. This is one is not among the recommendations. I think I know why.

DONALD R HUMPHREYS - See all my reviews

2.0 out of 5 stars Wordy, a bit pendantic and less useful than I had hoped, February 23, 2012

I agree with Antonio's review when he said, "I guess the main problem with the book is that you are interested in one particular area to use Python, say web development, or interfacing with databases this book would probably have insufficient detail, and you would want a specialist book in that area. Also I found the authors writing style somewhat verbose. Another issue is that those people who want to build a GUI for instance may not be interested in his choice of tool Tkinter."

From my perspective, this is another book that is way too wordy and one that seems to be an example of why programmers should probably not write texts that are meant to be tutorials.

My favorite author is Larry Ullman and after reading several of his books (about PHP/Mysql), I am finding that there is a lack of well-written books about Python in general in that they don't meet the standard of a text that help you learn a language and then put it to use with very little fuss or detours into arcane matters.

As the other reviewer noted, why does the author place an emphasis on teaching Tkinter which in my view is dated? It seems to be because as an author who is true to the 'Python way,' he defaults to teaching things that are core to the standard library/distribution of the language. I would think it would make sense to spend more time or even equal time on explaining a visual tool such as PyQT that is more friendly, state of the art and that allows for greater productivity.

Also, after investing in this book and a few other Python books because my interests includes GUI programming in general and with Python, I'm learning that these authors are not doing a good job of explaining the pros and cons of what's involved in distributing Python GUI Programs. Evidently, according to many forum entries, attempting to create and then distribute stand-alone Python Gui-based programs is a big deal compared with other options.

I still think the Python Language has the potential to be helpful to me - perhaps in console mode - but the Python books that I've read so far - and I chose authoritative sources - seem to be directed less to practical-minded users and more towards those who have the time for delving into things without asking "what is this useful for."

I have learned that for GUI program development, I will probably be better off using a VS product such as C# for GUI's, Perl for text file processing/editing and some CGI work and PHP for web site applications.

Paul A. Caskey on March 31, 2014

This book weighs FIVE POUNDS.

This is, by far, the biggest O'Reilly book I have ever seen. Maybe there is some Java book that matches it; I don't know. This one weighs 4 lbs 14 oz, and is almost 3 inches thick. Here is what you should do, if you buy this book:

1. Get a hacksaw and cut through the binding at page 355. Now you have a 3/4" thick book, from the front, containing a deep "introduction" to Python. This nice little rambling tutorial will be too confusing for a beginner, incomplete enough to be worthless as a reference, but very good if you are a PhD Computer Scientist interested in theoretical Object Oriented design, Python Internals, and a particularly confusing dive into python data structures. And parsing Windows directory trees. Read this little book once, and then chuck it into your nearest recycling bin.

2. Make your next hacksaw cut through the binding at page 768. This, oddly enough, produces another 3/4" thick book. Seal the binding with electrical tape. Label this book "Python/Tk GUI Programming" and stick in on your book shelf to collect dust. Reach for it some Sunday you are feeling nostalgic for the days when anyone cared about raw Windows or Linux GUI interfaces, instead of web interfaces.

3. What you have left is a hefty 830-page (!) O'Reilly book on Programming Python. This is the second half of the original book. This will now be on par with the other O'Reilly standards on Java or Perl already on your bookshelf -- measured by pure dead tree weight. This trimmed-down volume is a nice tome on Python client/server programming, Internet protocols, threads, textual data parsing theory and examples, database connections, and still some more Tk GUI stuff (the author can't seem to resist).

The 2010 publication date is a paradox, because this book only covers the new Python v3, which is a major split from Python 2. But every desktop and server in my work environment has Python 2.6 or 2.7 installed, so that's what I'm using. As a professional needing to come up to speed on Python, I need a clean examination of both Python 2 and 3. Certainly there is room for that in a 1600-page book, right? Apparently not. Plus, as a V3 reference, there are gaps in this book because it was published before Python 3 was fully baked.

So again, this book is a poor fit. No matter how you slice it. (rim shot)

Donate this book to a library, school, or sell it at a used book store. Whatever you do, don't pay to ship this beast back to Amazon. This shipping cost will kill you. Get ready for jaw drops from the guys at your local monthly programming group. If nothing else, this book is good --- for some laughs.

[Dec 25, 2010] Linux Developers choose Python as Best Programming Language and ...

Such polls mainly reflect what industry is using, no so much the quality of the language. In other poll the best Linux distribution is Ubuntu, which is probably the most primitive among the major distributions available.
According to Linux Journal readers, Python is both the best programming language and the best scripting language out there. This year, more than 12,000 developers on weighed in on what tools are helping them work and play as part of the Linux Journal's 2010 Readers' Choice Award - and it came as no surprise to those of us at ActiveState that Python came out on top as both the Best Scripting Language (beating out PHP, bash, PERL and Ruby) - and for the second straight year, Python also won as the Best Programming Language, once again edging out C++, Java, C and Perl for the honors.

At ActiveState, we continue see a steady stream of ActivePython Community Edition downloads, more enterprise deployments of ActivePython Business Edition, and a steady increase in the number of enterprise-ready Python packages in our PyPM Index that are being used by our customers over a wide range of verticals including high-tech, financial services, healthcare, and aerospace companies. Python has matured into an enterprise-class programming language that continues to nuture it's scripting world roots. We're happy to see Python get the recognition that it so justly deserves!

[Dec 25, 2010] Russ's Notes on Python

"I'm not entirely sure why Python has never caught on with me as a language to use on a regular basis. Certainly, one of the things that always bugs me is the lack of good integrated documentation support like POD (although apparently reStructured Text is slowly becoming that), but that's not the whole story. I suspect a lot is just that I'm very familiar with Perl and with its standard and supporting library, and it takes me longer to do anything in Python. But the language just feels slightly more awkward, and I never have gotten comfortable with the way that it uses exceptions for all error reporting. "

On the subject of C program indentation: In My Egotistical Opinion, most people's C programs should be indented six feet downward and covered with dirt.

- Blair P. Houghton

Introduction

Around the beginning of April, 2001, I finally decided to do something about the feeling I'd had for some time that I'd like to learn a few new programming languages. I started by looking at Python. These are my notes on the process.

Non-religious comments are welcome. Please don't send me advocacy.

I chose Python as a language to try (over a few other choices like Objective Caml or Common Lisp) mostly because it's less of a departure from the languages that I'm already comfortable with. In particular, it's really quite a bit like Perl. I picked this time to start since I had an idea for an initial program to try writing in Python, a program that I probably would normally write in Perl. I needed a program to help me manage releases of the various software package that I maintain, something to put a new version on an ftp site, update a series of web pages, generate a change log in a nice form for the web, and a few other similar things.

I started by reading the Python tutorial off www.python.org, pretty much straight through. I did keep an interactive Python process running while I did, but I didn't type in many of the examples; the results were explained quite well in the tutorial, and I generally don't need to do things myself to understand them. The tutorial is exceptionally well-written; after finishing reading it straight through (which took me an evening) and skimming the library reference, I felt I had a pretty good grasp on the language.

Things that immediately jumped out at me that I liked a lot:

There were a few things that I immediately didn't like, after having just read the tutorial:

There were also a couple of things that I immediately missed from other languages:

2001-04-08

Over the next few days, I started reading the language manual straight through, as well as poking around more parts of the language reference and writing some code. I started with a function to find the RCS keywords and version string in a file and from that extract the version and the last modified date (things that would need to be modified on the web page for that program). I really had a lot of fun with this.

The Python standard documentation is excellent. I mean truly superb. I can't really compare it to Perl (the other language that has truly excellent standard documentation), since I know Perl so well that I can't evaluate its documentation from the perspective of the beginner, but Python's tutorial eased me into the language beautifully and the language manual is well-written, understandable, and enjoyable to read. The library reference is well-organized and internally consistent, and I never had much trouble finding things. And they're available in info format as well as web pages, which is a major advantage for me; info is easier for me to read straight through, and web pages are easier for me to browse.

The language proved rather fun to write. Regex handling is a bit clunky since it's not a language built-in, but I was expecting that and I don't really mind it. The syntax is fun, and XEmacs python-mode does an excellent job handling highlighting and indentation. I was able to put together that little function and wrap a test around it fairly quickly (in a couple of hours while on the train, taking a lot of breaks to absorb the language reference manual or poke around in the library reference for the best way of doing something).

That's where I am at the moment. More as I find time to do more....

2001-05-04

I've finished my first Python program, after having gotten distracted by a variety of other things. It wasn't the program I originally started writing, since the problem of releasing a new version of a software package ended up being more complicated than I expected. (In particular, generating the documentation looks like it's going to be tricky.) I did get the code to extract version numbers and dates written, though, and then for another project (automatically generating man pages from scripts with embedded POD when installing them into our site-wide software installation) I needed that same code. So I wrote that program in Python and tested it and it works fine.

The lack of a way to safely execute a program without going through the shell is really bothering me. It was also the source of one of the three bugs in the first pass at my first program; I passed a multiword string to pod2man and forgot to protect it from the shell. What I'm currently doing is still fragile in the presence of single quotes in the string, which is another reason why I much prefer Perl's safe system() function. I feel like I must be missing something; something that fundamental couldn't possibly fail to be present in a scripting language.

A second bug in that program highlights another significant difference from Perl that I'm finding a little strange to deal with, namely the lack of equivalence between numbers and strings. My program had a dictionary of section titles, keyed by the section numbers, and I was using the plain number as the dictionary key. When I tried to look up a title in the dictionary, however, I used as the key a string taken from the end of the output filename, and 1 didn't match "1". It took me a while to track that down. (Admittedly, the problem was really laziness on my part; given the existence of such section numbers as "1m" and "3f", I should have used strings as the dictionary keys in the first place.)

The third bug, for the record, was attempting to use a Perl-like construct to read a file (while line = file.readline():). I see that Python 2.1 has the solution I really want in the form of xreadlines, but in the meantime that was easy enough to recode into a test and a break in the middle of the loop.

The lack of a standard documentation format like Perl's POD is bothering me and I'm not sure what to do about it. I want to put the documentation (preferrably in POD, but I'm willing to learn something else that's reasonably simple) into the same file as the script so that it gets updated when the script does and doesn't get lost in the directory. This apparently is just an unsolved problem, unless I'm missing some great link to an embedded documentation technique (and I quite possibly am). Current best idea is to put a long triple-quoted string at the end of my script containing POD. Ugh.

I took a brief look at the standard getopt library (although I didn't end up using it), and was a little disappointed; one of the features that I really liked about Perl's Getopt::Long was its ability to just stuff either the arguments to options or boolean values into variables directly, without needing something like the long case statement that's a standard feature of main() in many C programs. Looks like Python's getopt is much closer to C's, and requires something quite a bit like that case statement.

Oh, and while the documentation is still excellent, I've started noticing a gap in it when it comes to the core language (not the standard library; the documentation there is great). The language reference manual is an excellent reference manual, complete with clear syntax descriptions, but is a little much if one just wants to figure out how to do something. I wasn't sure of the syntax of the while statement, and the language reference was a little heavier than was helpful. I find myself returning to the tutorial to find things like this, and it has about the right level of explanation, but the problem with that is that the tutorial is laid out as a tutorial and isn't as easy to use as a reference. (For example, the while statement isn't listed in the table of contents, because it was introduced in an earlier section with a more general title.)

I need to get the info pages installed on my desktop machine so that I can look things up in the index easily; right now, I'm still using the documentation on the web.

2001-11-13

I've unfortunately not had very much time to work on this, as one can tell from the date.

Aahz pointed out a way to execute a program without going through the shell, namely os.spawnv(). That works, although the documentation is extremely poor. (Even in Python 2.1, it refers me to the Visual C++ Runtime Library documentation for information on what spawnv does, which is of course absurd.) At least the magic constants that it needs are relatively intuitive. Unfortunately, spawnv doesn't search the user's PATH for a command, and there's nothing like spawnvp. Sigh.

There's really no excuse for this being quite this hard. Executing a command without going through the shell is an extremely basic function that should be easily available in any scripting language without jumping through these sorts of hoops.

But this at least gave me a bit of experience in writing some more Python (a function to search the PATH to find a command), and the syntax is still very nice and convenient. I'm bouncing all over the tutorial and library reference to remember how to do things, but usually my first guesses are right.

I see that Debian doesn't have the info pages, only the HTML documentation. That's rather annoying, but workable. I now have the HTML documentation for Python 2.1 on local disk on my laptop.

2002-07-20

I've now written a couple of real Python programs (in addition to the simple little thing to generate man pages by running pod2man). You can find them (cvs2xhtml and cl2xhtml) with my web tools. They're not particularly pretty, but they work, and I now have some more experience writing simple procedural Python code. I still haven't done anything interesting with objects. Comments on the code are welcome. Don't expect too much.

There are a few other documentation methods for Python, but they seem primarily aimed at documenting modules and objects rather than documenting scripts. Pydoc in particular looks like it would be nice for API documentation but doesn't really do anything for end-user program documentation. Accordingly, I've given up for the time being on finding a more "native" approach and am just documenting my Python programs the way that I document most things, by writing embedded POD. I've yet to find a better documentation method; everything else seems to either be far too complicated and author-unfriendly to really write directly in (like DocBook) or can't generate Unix man pages, which I consider to be a requirement.

The Python documentation remains excellent, if scattered. I've sometimes spent a lot of time searching through the documentation to find the right module to do something, and questions of basic syntax are fairly hard to resolve (the tutorial is readable but not organized as a reference, and the language reference is too dense to provide a quick answer).

2004-03-03

My first major Python application is complete and working (although I'm not yet using it as much as I want to be using it). That's Tasker, a web-based to-do list manager written as a Python CGI script that calls a Python module.

I've now dealt with the Python module building tools, which are quite nice (nicer in some ways than Perl's Makefile.PL system with some more built-in functionality, although less mature in a few ways). Python's handling of the local module library is clearly less mature than Perl, and Debian's Python packages don't handle locally installed modules nearly as well as they should, but overall it was a rather positive experience. Built-in support for generating RPMs is very interesting, since eventually I'd like to provide .deb and RPM packages for all my software.

I played with some OO design for this application and ended up being fairly happy with how Python handled things. I'm not very happy with my object layout, but that's my problem, not Python's. The object system definitely feels far smoother and more comfortable to me than Perl's, although I can still write OO code faster in Perl because I'm more familiar with it. There's none of the $self hash nonsense for instance variables, though, which is quite nice.

The CGI modules for Python, and in particular the cgitb module for displaying exceptions nicely in the browser while debugging CGI applications, are absolutely excellent. I was highly impressed, and other than some confusion about the best way to retrieve POST data that was resolved after reading the documentation more closely, I found those modules very easy to work. The cgitb module is a beautiful, beautiful thing and by itself makes me want to use Python for all future CGI programming.

I still get caught all the time by the lack of interchangability of strings and numbers and I feel like I'm casting things all the time. I appreciate some of the benefits of stronger typing, but this one seems to get in my way more often than it helps.

I'm also still really annoyed at the lack of good documentation for the parts of the language that aren't considered part of the library. If I want documentation on how print works, I have only the tutorial and the detailed language standard, the former of which is not organized for reference and the latter of which is far too hard to understand. This is a gaping hole in the documentation that I really wish someone would fix. Thankfully, it only affects a small handful of things, like control flow constructs and the print statement, so I don't hit this very often, but whenever I do it's extremely frustrating.

I've given up on documentation for scripts and am just including a large POD section at the end of the script, since this seems to be the only option that will generate good man pages and good web pages. I'm not sure what to do about documentation for the module; there seem to be a variety of different proposals but nothing that I can really just use.

Oh, and one last point on documentation: the distutils documentation needs some work. Thankfully I found some really good additional documentation on the PyPI web site that explained a lot more about how to write a setup.py script.

2010-03-25

Six years later, I still find Python an interesting language, but I never got sufficiently absorbed by it for it to be part of my standard toolkit.

I've subsequently gotten some additional experience with extending Python through incorporating an extension written by Thomas Kula into the remctl distribution. The C interface is relatively nice and more comfortable than Perl, particularly since it doesn't involve a pseudo-C that is run through a preprocessor. It's a bit more comfortable to read and write.

Python's installation facilities, on the other hand, are poor. The distutils equivalent of Perl's ExtUtils::MakeMaker is considerably worse, despite ExtUtils::MakeMaker being old and crufty and strange. (I haven't compared it with Module::Build.) The interface is vaguely similar, but I had to apply all sorts of hacks to get the Python extension to build properly inside a Debian packaging framework, and integrating it with a larger package requires doing Autoconf substitution on a ton of different files. It was somewhat easier to avoid embedding RPATH into the module, but I'd still much rather work with Perl's facilities.

Similarly, while the test suite code has some interesting features (I'm using the core unittest framework), it's clearly inferior to Perl's Test::More support library and TAP protocol. I'm, of course, a known fan of Perl's TAP testing protocol (I even wrote my own implementation in C), but that's because it's well-designed, full-featured, and very useful. The Python unittest framework, by comparison, is awkward to use, has significantly inferior reporting capabilities, makes it harder to understand what test failed and isolate the failure, and requires a lot of digging around to understand how it works. I do like the use of decorators to handle skipping tests, and there are some interesting OO ideas around test setup and teardown, but the whole thing is more awkward than it should be.

I'm not entirely sure why Python has never caught on with me as a language to use on a regular basis. Certainly, one of the things that always bugs me is the lack of good integrated documentation support like POD (although apparently reStructured Text is slowly becoming that), but that's not the whole story. I suspect a lot is just that I'm very familiar with Perl and with its standard and supporting library, and it takes me longer to do anything in Python. But the language just feels slightly more awkward, and I never have gotten comfortable with the way that it uses exceptions for all error reporting.

I may get lured back into it again at some point, though, since Python 3.0 seems to have some very interesting features and it remains popular with people who know lots of programming languages. I want to give it another serious look with a few more test projects at some point in the future.

[Oct 09, 2010] How to Think Like a (Python) Programmer by Allen B. Downey

free e-book
www.diveintopython.net

Version 0.9.2

[Oct 09, 2010] free e-book

How to Think Like a (Python) Programmer

Allen B. Downey

Version 0.9.2

[Jan 15, 2010] The Quick Python Book, Second Edition by Vern Ceder

www.amazon.com

Paperback: 400 pages
Publisher: Manning Publications; 2nd edition (January 15, 2010)
Language: English
ISBN-10: 193518220X
ISBN-13: 978-1935182207
Product Dimensions: 7.4 x 0.7 x 9.2 inches

Alexandros Gezerlis "Alex Gezerlis" (Seattle, WA)(REAL NAME) - See all my reviews

Probably the best book on Python 3 currently available, July 10, 2010

"The Quick Python Book, Second Edition" is Vernon Ceder's reworking of the well-received volume "The Quick Python Book" by Daryl Harms and Kenneth McDonald. Ceder has removed a number of specialized chapters on COM, C & C++ extensions, JPython, HTMLgen & Zope and, more important, he has brought the text completely up to date, covering Python 3.1.

Most Python texts out there describe Python 2.x, so this book's main competition is: a) Mark Summerfield's "Programming in Python 3: A complete introduction to the Python Language, Second Edition", and b) Mark Pilgrim's "Dive into Python 3", while two other major books have incorporated material on Python 3, namely c) James Payne's "Beginning Python: Using Python 2.6 and Python 3.1" and d) Mark Lutz's "Learning Python: Powerful Object-Oriented Programming, 4th Edition".

The Good: this book is nice and short. It assumes a certain level of competence/background, so it does not waste space introducing the language-independent basics of flow control, object orientation, exception handling, and so on. It is example-based, and unlike in Pilgrim's volume the first few examples are short and thus readable. Chapter 3 ("The Quick Python overview") can be used as a compact reference when you're done reading the book, and various tables throughout the book help it function as a reference. Unlike its competition, it doesn't spend chapter upon chapter on databases, networking, or web applications. Instead, such topics are covered in only one (short) chapter at the end of the book. Ceder offers useful advice on the interrelation between older and newer Python features, whether discussing how to be more idiomatic (e.g. in chapter 6 on the format method vs % formatting, and in chapter 14 when introducing the "with" statement) or how to migrate from Python 2 to Python 3 (he devotes chapter 22 to this topic). On the publisher's website you can find a list of errata as well as the complete source code for the book. There you will see a link to an "Author online" forum in which you can interact with Ceder; perhaps more important, everyone who buys a paper copy of the book may also download a free PDF version. It is to be hoped that other publishers will follow Manning's example.

The Bad: the author is very clear that the book is aimed at those with experience in another programming language. Even so, in a few cases the assumptions are Python-specific (and hence unwarranted): one example is in chapter 5, where he lets us know that if x is a list then y=x[:] makes a copy of x, though this does not really explain why we cannot simply say y=x to accomplish the same goal. Another example: in chapter 12 Ceder uses character ranges expressed with [], though these are introduced much later (in chapter 17). Similarly, chapter 3 is quite good if you've already come into contact with Python before (even fleetingly). If you haven't, it may be obfuscating (though you could always just skip it on the first read). On a different note, this book does not contain exercises, though Summerfield's, Payne's, and Lutz's volumes do (along with answers). As mentioned in the previous paragraph, Ceder does not include too much extraneous stuff something which in my opinion is definitely a plus. However, he does not say absolutely anything on threading while Summerfield has a chapter on the subject and Payne a section. Similarly, Ceder does not mention function annotations at all, while Summerfield and Lutz each have a section on them. Finally, Ceder keeps referring the reader to the Python documentation for more details, and this can get frustrating. On the other hand, I suppose it would have been impossible for the book to stay at its current 320 pages otherwise.

Ceder's writing is concise, but this does not imply that he covers only the bare minimum of material. To pick a relatively advanced topic as an example, Ceder spends 2 pages on metaclasses, Summerfield 4.5 pages, Pilgrim and Payne devote half a page each only in the context of the changes from Python 2 to 3, while Lutz, in keeping with the mammoth size of his book, spends more than 30 pages on the topic. This (arbitrarily chosen) example is in some ways indicative of the wider approaches taken by the various Python 3 book authors.

In a nutshell, the fact that this book is considerably shorter than its competitors does not mean that it is shallow. The compactness is due partly to the author's succinct style of writing (which is not opaque, however) and partly to the fact that it does not contain too much on database programming, web services, and so on. All in all, if you're looking for a solid book on Python 3 that you stand a reasonable chance of reading cover-to-cover, then this is the volume you should buy. Four and a half stars.

Alex Gezerlis

[Dec 25, 2009] Bioinformatics Programming Using Python Practical Programming for Biological Data by Mitchell L. Model

Notable quotes:
"... Comparing to Perl, Python has a quite lagged adoption as the scripting language of choice in the field of bioinformatics, although it is getting some moment recently. If you read job descriptions for bioinformatics engineer or scientist positions a few year back, you barely saw Python mentioned, even as "nice to have optional skill". ..."
"... Moreover, it can actually serve as a good introductory book to Python regardless the main focus on bioinformatics examples. ..."
www.amazon.com

Paperback: 528 pages
Publisher: O'Reilly Media; 1st edition (December 25, 2009)
Language: English
ISBN-10: 059615450X
ISBN-13: 978-0596154509
Product Dimensions: 7 x 1 x 9.2 inches

C. Chin on February 15, 2010

Good introductory book for learning both bioinformatics and python

Comparing to Perl, Python has a quite lagged adoption as the scripting language of choice in the field of bioinformatics, although it is getting some moment recently. If you read job descriptions for bioinformatics engineer or scientist positions a few year back, you barely saw Python mentioned, even as "nice to have optional skill". One of the reasons is probably lacking of good introductory level bioinformatics books in Python so there are, in general, less people thinking Python as a good choice for bioinformatics. The book "Beginning Perl for Bioinformatics" from O Reilly was published in 2001. Almost one decade later, we finally get the book "Bioinformatics Programming Using Python" from Mitchell Model to fill the gap.

When I first skimmed the book "Bioinformatics Programming Using Python", I got the impression that this book was more like "learning python using bioinformatics as examples" and felt a little bit disappointed as I was hoping for more advanced content. However, once I went through the book, reading the preface and everything else chapter by chapter, I understood the main target audiences that author had in mind and I thought the author did a great job in fulfilling the main purpose.

In modern biological research, scientists can easily generate large amount of data where Excel spreadsheets that most bench scientists use to process limiting amount of data is no longer an option. I personally believe that the new generation of biologists will have to learn how to process and manage large amount inhomogeneous data to make new discovery out of it. This requires general computational skill beyond just knowing how to use some special purpose applications that some software vendor can provide. The book gives good introduction about practical computational skills using Python to process bioinformatics data. The book is very well organized for a newbie who just wants to start to process the raw data their own and get into a process of learning-by-doing to become a Python programmer.

The book starts with an introduction on the primitive data types in Python and moves toward the flow controls and collection data type with emphasis on, not surprisingly, string processing and file parsing, two of most common tasks in bioinformatics. Then, the author introduces the object-oriented programming in Python. I think a beginner will also like those code templates for different patterns of data processing task in Chapter 4. They summarize the usual flow structure for common tasks very well.

After giving the basic concept of programming with Python, the author focuses on other utilities which are very useful for day-to-day work for gathering, extracting, and processing data from different data sources. For example, the author discusses about how to explore and organize files with Python in the OS level, using regular expression for extracting complicated text data file, XML processing, web programming for fetching online biological data and sharing data with a simple web server, and, of course, how to program Python to interact with a database. The deep knowledge of all of these topics might deserve their own books. The author does a good job to cover all these topics in a concise way. This will help people to know what can be done very easily with Python and, if they want, to learn any of those topic more from other resources. The final touch of the book is on structured graphics. This is very wise choice since the destiny of most of bioinformatics data is very likely to be some graphs used in presentations and for publishing. Again, there are many other Python packages can help scientists to generate nice graph, but the author focuses on one or two of them to show the readers how to do general some graphs with them and the reader might be able to learn something else from there.

One thing I hope the author can also cover, at least at a beginner level, is the numerical and statistical aspect in bioinformatics computing with Python. For example, Numpy or Scipy are very useful for processing large amount of data, generating statistics and evaluating significance of the results. They are very useful especially for processing large amount data where the native Python objects are no longer efficient enough. The numerical computation aspect in bioinformatics is basically lacking in the book. The other thing that might be desirable for such a book is to show that Python is a great tool for prototyping some algorithms in bioinformatics. This is probably my own personal bias, but I do think it is nice to show some basic bioinformatics algorithm implementations in python. This will help the readers to understand a little bit more about some of the common algorithms used in the field and to get a taste on a little bit more advanced programming.

Overall, I will not hesitate to recommend this book to any one who will like to start to process biological data on their own with Python. Moreover, it can actually serve as a good introductory book to Python regardless the main focus on bioinformatics examples. The book covers most day-to-day basic bioinformatics tasks and shows Python is a great tool for those tasks.

I think a little more advanced topics, especially on basic numerical and statistical computation in the book, will also help the target audiences. Unfortunately, none of that topic is mentioned in the book. That has been said, even if you are an experienced python programmer in bioinformatics, the book's focus on Python 3 and a lot of useful templates might serve well as a quick reference if you are looking for something you do not have direct experience before.

[Aug 18, 2009] THE NEXT SNAKE BY RAINER GRIMM

What do Python 2.x programmers need to know about Python 3?

With the latest major Python release, creator Guido van Rossum saw the opportunity to tidy up his famous scripting language. What is different about Python 3.0? In this article, I offer some highlights for Python programmers who are thinking about making the switch to 3.x.

Read full article as PDF "

[Jun 20, 2009] A Python Client/Server Tutorial by Phillip Watts

June 16, 2009

There can be many reasons why you might need a client/server application. For a simple example, purchasing for a small retail chain might need up to the minute stock levels on a central server. The point-of-sale application in the stores would then need to post inventory transactions to the central server in real-time.

This application can easily be coded in Python with performance levels of thousands of transactions per second on a desktop PC. Simple sample programs for the server and client sides are listed below, with discussions following.

[Jul 8, 2008] Python Call Graph 0.5.1 by Gerald Kaszuba

About: pycallgraph is a Python library that creates call graphs for Python programs.

Changes: The "pycg" command line tool was renamed to "pycallgraph" due to naming conflicts with other packages.

[Jun 23, 2008] freshmeat.net Project details for cfv

About:
cfv is a utility to both test and create .sfv (Simple File Verify), .csv, .crc, .md5(sfv style), md5sum, BSD md5, sha1sum, and .torrent checksum verification files. It also includes test-only support for .par and .par2 files. These files are commonly used to ensure the correct retrieval or storage of data.

Release focus: Major bugfixes

Changes:
Help output is printed to stdout under non-error conditions. A mmap file descriptor leak in Python 2.4.2 was worked around. The different module layout of BitTorrent 5.x is supported. A "struct integer overflow masking is deprecated" warning was fixed. The --private_torrent flag was added. A bug was worked around in 64-bit Python version 2.5 and later which causes checksums of files larger than 4GB to be incorrectly calculated when using mmap.

[Jun 20, 2008] BitRock Download Web Stacks

BitRock Web Stacks provide you with the easiest way to install and run the LAMP platform in a variety of Linux distributions. BitRock Web Stacks are free to download and use under the terms of the Apache License 2.0. To learn more about our licensing policies, click here.

You can find up-to-date WAMP, LAMP and MAMP stacks at the BitNami open source website. In addition to those, you will find freely available application stacks for popular open source software such as Joomla!, Drupal, Mediawiki and Roller. Just like BitRock Web Stacks, they include everything you need to run the software and come packaged in a fast, easy to use installer.

BitRock Web Stacks contain several open source tools and libraries. Please be sure that you read and comply with all of the applicable licenses. If you are a MySQL Network subscriber (or would like to purchase a subscription) and want to use a version of LAMPStack that contains the MySQL Certified binaries, please send an email to sales@bitrock.com.

For further information, including supported platforms, component versions, documentation, and support, please visit our solutions section.

[Mar 12, 2008] Terminator - Multiple GNOME terminals in one window

Rewrite of screen in Python?

This is a project to produce an efficient way of filling a large area of screen space with terminals. This is done by splitting the window into a resizeable grid of terminals. As such, you can produce a very flexible arrangements of terminals for different tasks.

Read me

Terminator 0.8.1
by Chris Jones <cmsj@tenshu.net>

This is a little python script to give me lots of terminals in a single window, saving me valuable laptop screen space otherwise wasted on window decorations and not quite being able to fill the screen with terminals.

Right now it will open a single window with one terminal and it will (to some degree) mirror the settings of your default gnome-terminal profile in gconf. Eventually this will be extended and improved to offer profile selection per-terminal, configuration thereof and the ability to alter the number of terminals and save meta-profiles.

You can create more terminals by right clicking on one and choosing to split it vertically or horizontally. You can get rid of a terminal by right clicking on it and choosing Close. ctrl-shift-o and ctrl-shift-e will also effect the splitting.

ctrl-shift-n and ctrl-shift-p will shift focus to the next/previous terminal respectively, and ctrl-shift-w will close the current terminal and ctrl-shift-q the current window

Ask questions at: https://answers.launchpad.net/terminator/
Please report all bugs to https://bugs.launchpad.net/terminator/+filebug

It's quite shamelessly based on code in the vte-demo.py from the vte widget package, and on the gedit terminal plugin (which was fantastically useful).

vte-demo.py is not my code and is copyright its original author. While it does not contain any specific licensing information in it, the VTE package appears to be licenced under LGPL v2.

the gedit terminal plugin is part of the gedit-plugins package, which is licenced under GPL v2 or later.

I am thus licensing Terminator as GPL v2 only.

Cristian Grada provided the icon under the same licence.

Python and the Programmer

Python and the Programmer
A Conversation with Bruce Eckel, Part I
by Bill Venners
Jun 2, 2003
Summary
Bruce Eckel talks with Bill Venners about why he feels Python is "about him," how minimizing clutter improves productivity, and the relationship between backwards compatibility and programmer pain.

Bruce Eckel wrote the best-selling books Thinking in C++ and Thinking in Java, but for the past several years he's preferred to think in Python. Two years ago, Eckel gave a keynote address at the 9th International Python Conference entitled "Why I love Python." He presented ten reasons he loves programming in Python in "top ten list" style, starting with ten and ending with one.

In this interview, which is being published in weekly installments, I ask Bruce Eckel about each of these ten points. In this installment, Bruce Eckel explains why he feels Python is "about him," how minimizing clutter improves productivity, and the relationship between backwards compatibility and programmer pain.

Bill Venners: In the introduction to your "Why I Love Python" keynote, you said what you love the most is "Python is about you." How is Python about you?

Bruce Eckel: With every other language I've had to deal with, it's always felt like the designers were saying, "Yes, we're trying to make your life easier with this language, but these other things are more important." With Python, it has always felt like the designers were saying, "We're trying to make your life easier, and that's it. Making your life easier is the thing that we're not compromising on."

For example, the designers of C++ certainly attempted to make the programmer's life easier, but always made compromises for performance and backwards compatibility. If you ever had a complaint about the way C++ worked, the answer was performance and backwards compatibility.

Bill Venners: What compromises do you see in Java? James Gosling did try to make programmers more productive by eliminating memory bugs.

Bruce Eckel: Sure. I also think that Java's consistency of error handling helped programmer productivity. C++ introduced exception handling, but that was just one of many ways to handle errors in C++. At one time, I thought that Java's checked exceptions were helpful, but I've modified my view on that. (See Resources.)

It seems the compromise in Java is marketing. They had to rush Java out to market. If they had taken a little more time and implemented design by contract, or even just assertions, or any number of other features, it would have been better for the programmer. If they had done design and code reviews, they would have found all sorts of silliness. And I suppose the way Java is marketed is probably what rubs me the wrong way about it. We can say, "Oh, but we don't like this feature," and the answer is, "Yes, but, marketing dictates that it be this way."

Maybe the compromises in C++ were for marketing reasons too. Although choosing to be efficient and backwards compatible with C was done to sell C++ to techies, it was still to sell it to somebody.

I feel Python was designed for the person who is actually doing the programming, to maximize their productivity. And that just makes me feel warm and fuzzy all over. I feel nobody is going to be telling me, "Oh yeah, you have to jump through all these hoops for one reason or another." When you have the experience of really being able to be as productive as possible, then you start to get pissed off at other languages. You think, "Gee, I've been wasting my time with these other languages."

Number 10: Reduced Clutter

Bill Venners: In your keynote, you gave ten reasons you love Python. Number ten was reduced clutter. What did you mean by reduced clutter?

Bruce Eckel: They say you can hold seven plus or minus two pieces of information in your mind. I can't remember how to open files in Java. I've written chapters on it. I've done it a bunch of times, but it's too many steps. And when I actually analyze it, I realize these are just silly design decisions that they made. Even if they insisted on using the Decorator pattern in java.io, they should have had a convenience constructor for opening files simply. Because we open files all the time, but nobody can remember how. It is too much information to hold in your mind.

The other issue is the effect of an interruption. If you are really deep into doing something and you have an interruption, it's quite a number of minutes before you can get back into that deeply focused state. With programming, imagine you're flowing along. You're thinking, "I know this, and I know this, and I know this," and you are putting things together. And then all of a sudden you run into something like, "I have to open a file and read in the lines." All the clutter in the code you have to write to do that in Java can interrupt the flow of your work.

Another number that used to be bandied about is that programmers can produce an average of ten working lines of code per day. Say I open up a file and read in all the lines. In Java, I've probably already used up my ten working lines of code for that day. In Python, I can do it in one line. I can say, "for line in file('filename').readlines():," and then I'm ready to process the lines. And I can remember that one liner off the top of my head, so I can just really flow with that.

Python's minimal clutter also helps when I'm reading somebody else's code. I'm not tripping over verbose syntax and idioms. "Oh I see. Opening the file. Reading the lines." I can grok it. It's very similar to the design patterns in that you have a much denser form of communication. Also, because blocks are denoted by indentation in Python, indentation is uniform in Python programs. And indentation is meaningful to us as readers. So because we have consistent code formatting, I can read somebody else's code and I'm not constantly tripping over, "Oh, I see. They're putting their curly braces here or there." I don't have to think about that.

Number 9: Not Backwards Compatible in Exchange for Pain

Bill Venners: In your keynote, your ninth reason for loving Python was, "Not backwards compatible in exchange for pain." Could you speak a bit about that?

Bruce Eckel: That's primarily directed at C++. To some degree you could say it refers to Java because Java was derived primarily from C++. But C++ in particular was backwards compatible with C, and that justified lots of language issues. On one hand, that backwards compatibility was a great benefit, because C programmers could easily migrate to C++. It was a comfortable place for C programmers to go. But on the other hand, all the features that were compromised for backwards compatibility was the great drawback of C++.

Python isn't backwards compatible with anything, except itself. But even so, the Python designers have actually modified some fundamental things in order to fix the language in places they decided were broken. I've always heard from Sun that backwards compatibility is job one. And so even though stuff is broken in Java, they're not going to fix it, because they don't want to risk breaking code. Not breaking code always sounds good, but it also means we're going to be in pain as programmers.

One fundamental change they made in Python, for example, was "type class unification." In earlier versions, some of Python's primitive types were not first class objects with first class characteristics. Numbers, for example, were special cases like they are in Java. But that's been modified so now I can inherit from integer if I want to. Or I can inherit from the modified dictionary class. That couldn't be done before. After a while it began to be clear that it was a mistake, so they fixed it.

Now in C++ or Java, they'd say, "Oh well, too bad." But in Python, they looked at two issues. One, they were not breaking anybody's existing world, because anyone could simply choose to not upgrade. I think that could be an attitude taken by Java as well. And two, it seemed relatively easy to fix the broken code, and the improvement seemed worth the code-fixing work. I find that attitude so refreshing, compared to the languages I'd used before where they said, "Oh, it's broken. We made a mistake, but you'll have to live with it. You'll have to live with our mistakes."

Next Week

Come back Monday, June 9 for Part I of a conversation with Java's creator James Gosling. I am now staggering the publication of several interviews at once, to give the reader variety. The next installment of this interview with Bruce Eckel will appear on Monday, June 23. If you'd like to receive a brief weekly email announcing new articles at Artima.com, please subscribe to the Artima Newsletter.

Talk Back!

Have an opinion about programmer productivity, backwards compatibility, or breaking code versus programmer pain. Discuss this article in the News & Ideas Forum topic, Python and the Programmer.

Resources

Bruce Eckel's Mindview, Inc.:
http://www.mindview.net/

Bruce Eckel's essay on checked exceptions: Does Java Need Checked Exceptions?:
http://www.mindview.net/Etc/Discussions/CheckedExceptions

Bruce Eckel's Public and In-House Seminars:
http://mindview.net/Seminars

Bruce Eckel's Weblog:
http://www.mindview.net/WebLog

Python.org, the Python Language Website:
http://www.python.org/

Introductory Material on Python:
http://www.python.org/doc/Intros.html

Python Tutorial:
http://www.python.org/doc/current/tut/tut.html

Python FAQ Wizard:
http://www.python.org/cgi-bin/faqw.py

[Apr 3, 2007] Charming Python Python elegance and warts, Part 1

Generators as not-quite-sequences

Over several versions, Python has hugely enhanced its "laziness." For several versions, we have had generators defined with the yield statement in a function body. But along the way we also got the itertools modules to combine and create various types of iterators. We have the iter() built-in function to turn many sequence-like objects into iterators. With Python 2.4, we got generator expressions, and with 2.5 we will get enhanced generators that make writing coroutines easier. Moreover, more and more Python objects have become iterators or iterator-like; for example, what used to require the .xreadlines() method or before that the xreadlines module, is now simply the default behavior of open() to read files.

Similarly, looping through a dict lazily used to require the .iterkeys() method; now it is just the default for key in dct behavior. Functions like xrange() are a bit "special" in being generator-like, but neither quite a real iterator (no .next() method), nor a realized list like range() returns. However, enumerate() returns a true generator, and usually does what you had earlier wanted xrange() for. And itertools.count() is another lazy call that does almost the same thing as xrange(), but as a full-fledged iterator.

Python is strongly moving towards lazily constructing sequence-like objects; and overall this is an excellent direction. Lazy pseudo-sequences both save memory space and speed up operations (especially when dealing with very large sequence-like "things").

The problem is that Python still has a schizoaffective condition when it comes to deciding what the differences and similarities between "hard" sequences and iterators are. The troublesome part of this is that it really violates Python's idea of "duck typing": the ability to use a given object for a purpose just as long as it has the right behaviors, but not necessarily any inheritance or type restriction. The various things that are iterators or iterator-like sometimes act sequence-like, but other times do not; conversely, sequences often act iterator-like, but not always. Outside of those steeped in Python arcana, what does what is not obvious.

Divergences

The main point of similarity is that everything that is sequence- or iterator-like lets you loop over it, whether using a for loop, a list comprehension, or a generator comprehension. Past that, divergences occur. The most important of these differences is that sequences can be indexed, and directly sliced, while iterators cannot. In fact, indexing into a sequence is probably the most common thing you ever do with a sequence -- why on earth does it fall down so badly on iterators? For example:


Listing 9. Sequence-like and iterator-like things
>>> r = range(10)

>>> i = iter(r)

>>> x = xrange(10)

>>> g = itertools.takewhile(lambda n: n<10, itertools.count())

#...etc...

For all of these, you can use for n in thing. In fact, if you "concretize" any of them with list(thing), you wind up with exactly the same result. But if you wish to obtain a specific item -- or a slice of a few items -- you need to start caring about the exact type of thing. For example:


Listing 10. When indexing succeeds and fails
>>> r[4]
4

>>> i[4]
TypeError: unindexable object

With enough contortions, you can get an item for every type of sequence/iterator. One way is to loop until you get there. Another hackish combination might be something like:


Listing 11. Contortions to obtain an index
>>> thing, temp = itertools.tee(thing)

>>> zip(temp, '.'*5)[-1][0]
4

The pre-call to itertools.tee() preserves the original iterator. For a slice, you might use the itertools.islice() function, wrapped up in contortions.


Listing 12. Contortions to obtain a slice
>>> r[4:9:2]
[4, 6, 8]

>>> list(itertools.islice(r,4,9,2))  # works for iterators
[4, 6, 8]

A class wrapper

You might combine these techniques into a class wrapper for convenience, using some magic methods:


Listing 13. Making iterators indexable
>>> class Indexable(object):
...     def __init__(self, it):
...         self.it = it
...     def __getitem__(self, x):
...         self.it, temp = itertools.tee(self.it)
...         if type(x) is slice:
...             return list(itertools.islice(self.it, x.start, x.stop, x.step))
...         else:
...             return zip(temp, range(x+1))[-1][0]
...     def __iter__(self):
...         self.it, temp = itertools.tee(self.it)
...         return temp
...

>>> integers = Indexable(itertools.count())

>>> integers[4]
4
>>> integers[4:9:2]
[4, 6, 8]
      

So with some effort, you can coax an object to behave like both a sequence and an iterator. But this much effort should really not be necessary; indexing and slicing should "just work" whether a concrete sequence or a iterator is involved.

Notice that the Indexable class wrapper is still not as flexible as might be desirable. The main problem is that we create a new copy of the iterator every time. A better approach would be to cache the head of the sequence when we slice it, then use that cached head for future access of elements already examined. Of course, there is a trade-off between memory used and the speed penalty of running through the iterator. Nonetheless, the best thing would be if Python itself would do all of this "behind the scenes" -- the behavior might be fine-tuned somehow by "power users," but average programmers should not have to think about any of this.

In the next installment in this series, I'll discuss accessing methods using attribute syntax.

[Oct 26, 2006] ONLamp.com -- What's New in Python 2.5

It's clear that Python is under pressure from Ruby :-)

It's hard to believe Python is more than 15 years old already. While that may seem old for a programming language, in the case of Python it means the language is mature. In spite of its age, the newest versions of Python are powerful, providing everything you would expect from a modern programming language.

This article provides a rundown of the new and important features of Python 2.5. I assume that you're familiar with Python and aren't looking for an introductory tutorial, although in some cases I do introduce some of the material, such as generators.

[Sep 30, 2006] Python 2.5 Release We are pleased to announce the release of Python 2.5 (FINAL), the final, production release of Python 2.5, on September 19th, 2006.

with open('/etc/passwd', 'r') as f:
    for line in f:
        print line
        ... more processing code ...
why to stray from mainstream C-style is unlear to me. When developing computer language syntax, natural language imitation should not be the priority - also being different for the sake of being different is so very early 90s
cout << ( a==b ? "first option" : "second option" )

[Sept 20, 2006] Python 101 cheat sheet

[01 Feb 2000] Python columnist Evelyn Mitchell brings you a quick reference and learning tools for newbies who want to get to know the language. Print it, keep it close at hand, and get down to programming!

[Jul 27, 2006] eWeek/Microsoft Ships Python on .Net Darryl K. Taft

Microsoft has shipped the release candidate for IronPython 1.0 on its CodePlex community source site.

In a July 25 blog post, S. "Soma" Somasegar, corporate vice president of Microsoft's developer division, praised the team for getting to a release candidate for a dynamic language that runs on the Microsoft CLI (Common Language Infrastructure). Microsoft designed the CLI to support a variety of programming languages. Indeed, "one of the great features of the .Net framework is the Common Language Infrastructure," Somasegar said.

"IronPython is a project that implements the dynamic object-oriented Python language on top of the CLI," Somasegar said. IronPython is both well-integrated with the .Net Framework and is a true implementation of the Python language, he said.

And ".Net integration means that this rich programming framework is available to Python developers and that they can interoperate with other .Net languages and tools," Somasegar said. "All of Python's dynamic features like an interactive interpreter, dynamically modifying objects and even metaclasses are available. IronPython also leverages the CLI to achieve good performance, running up to 1.5 times faster than the standard C-based Python implementation on the standard Pystone benchmark."

Click here to read an eWEEK interview with Python creator Guido van Rossum.

Moreover, the download of the release candidate for IronPython 1.0 "includes a tutorial which gives .Net programmers a great way to get started with Python and Python programmers a great way to get started with .Net," Somasegar said.

Somasegar said he finds it "exciting to see that the Visual Studio SDK [software development kit] team has used the IronPython project as a chance to show language developers how they can build support for their language into Visual Studio. They have created a sample, with source, that shows some of the basics required for integrating into the IDE including the project system, debugger, interactive console, IntelliSense and even the Windows forms designer. "

IronPython is the creation of Jim Hugunin, a developer on the Microsoft CLR (Common Language Runtime) team. Hugunin joined Microsoft in 2004.

In a statement written in July 2004, Hugunin said: "My plan was to do a little work and then write a short pithy article called, 'Why .Net is a terrible platform for dynamic languages.' My plans changed when I found the CLR to be an excellent target for the highly dynamic Python language. Since then I've spent much of my spare time working on the development of IronPython."

However, Hugunin said he grew frustrated with the slow pace of progress he could make by working on the project only in his spare time, so he decided to join Microsoft.

IronPython is governed by Microsoft's Shared Source license.

Dig Deep into Python Internals by Gigi Sayfan Part 1 of 2

Python, the open source scripting language, has grown tremendously popular in the last five years-and with good reason. Python boasts a sophisticated object model that wise developers can exploit in ways that Java, C++, and C# developers can only dream of.

This article is the first in a two-part series that will dig deep to explore the fascinating new-style Python object model, which was introduced in Python 2.2 and improved in 2.3 and 2.4. The object model and type system are very dynamic and allow quite a few interesting tricks. In this article I will describe the object, model, and type system; explore various entities; explain the life cycle of an object; and introduce some of the countless ways to modify and customize almost everything you thought immutable at runtime.

The Python Object Model
Python's objects are basically a bunch of attributes. These attributes include the type of the object, fields, methods, and base classes. Attributes are also objects, accessible through their containing objects.

The built-in dir() function is your best friend when it comes to exploring python objects. It is designed for interactive use and, thereby, returns a list of attributes that the implementers of the dir function thought would be relevant for interactive exploration. This output, however, is just a subset of all the attributes of the object. The code sample below shows the dir function in action. It turns out that the integer 5 has many attributes that seem like mathematical operations on integers.


dir(5)

['__abs__', '__add__', '__and__', '__class__', '__cmp__', '__coerce__', '__delattr__', '__div__', 
'__divmod__', '__doc__', '__float__', '__floordiv__', '__getattribute__', '__getnewargs__', 
'__hash__', '__hex__', '__init__', '__int__', '__invert__', '__long__', '__lshift__', '__mod__',
'__mul__', '__neg__', '__new__', '__nonzero__', '__oct__', '__or__', '__pos__', '__pow__', '
__radd__', '__rand__', '__rdiv__', '__rdivmod__', '__reduce__', '__reduce_ex__', '__repr__',
'__rfloordiv__', '__rlshift__', '__rmod__', '__rmul__', '__ror__', '__rpow__', '__rrshift__',
'__rshift__', '__rsub__', '__rtruediv__', '__rxor__', '__setattr__', '__str__', '__sub__', '__truediv__', '__xor__']
The function foo has many attributes too. The most important one is __call__ which means it is a callable type. You do want to call your functions, don't you?

def foo()
      pass
...
dir(foo)

['__call__', '__class__', '__delattr__', '__dict__', '__doc__', '__get__', '__getattribute__',
'__hash__', '__init__', '__module__', '__name__', '__new__', '__reduce__', '__reduce_ex__',
'__repr__', '__setattr__', '__str__', 'func_closure', 'func_code', 'func_defaults', 'func_dict', 
'func_doc', 'func_globals', 'func_name']
Next I'll define a class called 'A' with two methods, __init__ and dump, and an instance field 'x' and also an instance 'a' of this class. The dir function shows that the class's attributes include the methods and the instance has all the class attributes as well as the instance field.

>>> class A(object):
...     def __init__(self):
...             self.x = 3
...     def dump(self):
...             print self.x
...
>>> dir(A)

['__class__', '__delattr__', '__dict__', '__doc__', '__getattribute__', '__hash__', '__init__', 
'__module__', '__new__', '__reduce__', '__reduce_ex__', '__repr__', '__setattr__', '__str__', 
'__weakref__', 'dump']

>>> a = A()
>>> dir(a)

['__class__', '__delattr__', '__dict__', '__doc__', '__getattribute__', '__hash__', '__init__',
'__module__', '__new__', '__reduce__', '__reduce_ex__', '__repr__', '__setattr__', '__str__',
'__weakref__', 'dump', 'x']

The Python Type System
Python has many types. Much more than you find in most languages (at least explicitly). This means that the interpreter has a lot of information at runtime and the programmer can take advantage of it by manipulating types at runtime. Most types are defined in the types module, which is shown in the code immediately below. Types come in various flavors: There are built-in types, new-style classes (derived from object), and old-style classes (pre Python 2.2). I will not discuss old-style classes since they are frowned upon by everybody and exist only for backward compatibility.


>>> import types
>>> dir(types)

['BooleanType', 'BufferType', 'BuiltinFunctionType', 'BuiltinMethodType', 'ClassType', 'CodeType',
'ComplexType', 'DictProxyType', 'DictType', 'DictionaryType', 'EllipsisType', 'FileType', 
'FloatType', 'FrameType', 'FunctionType', 'GeneratorType', 'InstanceType', 'IntType', 'LambdaType',
'ListType', 'LongType', 'MethodType', 'ModuleType', 'NoneType', 'NotImplementedType', 'ObjectType',
'SliceType', 'StringType', 'StringTypes', 'TracebackType', 'TupleType', 'TypeType', 
'UnboundMethodType', 'UnicodeType', 'XRangeType', '__builtins__', '__doc__', '__file__', '__name__']
Python's type system is object-oriented. Every type (including built-in types) is derived (directly or indirectly) from object. Another interesting fact is that types, classes and functions are all first-class citizens and have a type themselves. Before I delve down into some juicy demonstrations let me introduce the built-in function 'type'. This function returns the type of any object (and also serves as a type factory). Most of these types are listed in the types module, and some of them have a short name. Below I've unleashed the 'type' function on several objects: None, integer, list, the object type, type itself, and even the 'types' module. As you can see the type of all types (list type, object, and type itself) is 'type' or in its full name types.TypeType (no kidding, that's the name of the type).

>>> type(None)
<type 'NoneType'>

>>> type(5)
<type 'int'>

>>> x = [1,2,3]
>>> type(x)
<type 'list'>

>>> type(list)
<type 'type'> 

>>> type(type)
<type 'type'>

>>> type(object)
<type 'type'>
>>>

>>> import types
>>> type(types)
<type 'module'>

>>> type==types.TypeType
True
What is the type of classes and instances? Well, classes are types of course, so their type is always 'type' (regardless of inheritance). The type of class instances is their class.

>>> class A(object):
...     pass


>>> a = A()

>>> type(A)
<type 'type'>

>>> type(a)
<class '__main__.A'>

>>> a.__class__
<class '__main__.A'>
It's time for the scary part-a vicious cycle: 'type' is the type of object, but object is the base class of type. Come again? 'type' is the type of object, but object is the base class of type. That's right-circular dependency. 'object' is a 'type' and 'type' is an 'object'.

>>> type(object)
<type 'type'>

>>> type.__bases__
(<type 'object'>,)

>>> object.__bases__
()
How can it be? Well, since the core entities in Python are not implemented themselves in Python (there is PyPy but that's another story) this is not really an issue. The 'object' and 'type' are not really implemented in terms of each other.

The one important thing to take home from this is that types are objects and are therefore subject to all the ramifications thereof. I'll discuss those ramifications very shortly.

Instances, Classes, Class Factories, and Metaclasses
When I talk about instances I mean object instances of a class derived from object (or the object class itself). A class is a type, but as you recall it is also an object (of type 'type'). This allows classes to be created and manipulated at runtime. This code demonstrates how to create a class at runtime and instantiate it.


def init_method(self, x, y):
      self.x = x
      self.y = y
def dumpSum_method(self):
      print self.x + self.y

D = type('DynamicClass',
                   (object,),
                   {'__init__':init_method, 'dumpSum':dumpSum_method})
d = D(3, 4)
d.dumpSum()
As you can see I created two functions (init_method and dumpSum_method) and then invoked the ubiquitous 'type' function as a class factory to create a class called 'DynamicClass,' which is derived from 'object' and has two methods (one is the __init__ constructor).

It is pretty simple to create the functions themselves on the fly too. Note that the methods I attached to the class are regular functions that can be called directly (provided their self-argument has x and y members, similar to C++ template arguments).

Functions, Methods and other Callables
Python enjoys a plethora of callable objects. Callable objects are function-like objects that can be invoked by calling their () operator. Callable objects include plain functions (module-level), methods (bound, unbound, static, and class methods) and any other object that has a __call__ function attribute (either in its own dictionary, via one of its ancestors, or through a descriptor).

It's truly complicated so the bottom line is to remember that all these flavors of callables eventually boil down to a plain function. For example, in the code below the class A defines a method named 'foo' that can be accessed through:

  1. an instance so it is a bound method (bound implicitly to its instance)
  2. through the class A itself and then it is an unbound method (the instance must be supplied explicitly)
  3. directly from A's dictionary, in which case it is a plain function (but you must still call it with an instance of A).
So, all methods are actually functions but the runtime assigns different types depending on how you access it.

class A(object):
    def foo(self):
        print 'I am foo'

>>> a = A()
>>> a.foo
<bound method A.foo of <__main__.A object at 0x00A13EB0>>

>>> A.foo
<unbound method A.foo>

>>> A.__dict__['foo']
<function foo at 0x00A0A3F0>
>>> a.foo

>>> a.foo()
I am foo
>>> A.foo(a)
I am foo
>>> A.__dict__['foo'](a)
I am foo
Let's talk about static methods and class methods. Static methods are very simple. They are similar to static methods in Java/C++/C#. They are scoped by their class but they don't have a special first argument like instance methods or class methods do; they act just like a regular function (you must provide all the arguments since they can't access any instance fields). Static methods are not so useful in Python because regular module-level functions are already scoped by their module and they are the natural mapping to static methods in Java/C++/C#.

Class methods are an exotic animal. Their first argument is the class itself (traditionally named cls) and they are used primarily in esoteric scenarios. Static and class methods actually return a wrapper around the original function object. In the code that follows, note that the static method may be accessed either through an instance or through a class. The class method accepts a cls instance as its first argument but cls is invoked through a class directly (no explicit class argument). This is different from an unbound method where you have to provide an instance explicitly as first argument.


class A(object):
    def foo():
        print 'I am foo'
    def foo2(cls):
        print 'I am foo2', cls 
    def foo3(self):
        print 'I am foo3', self       
    foo=staticmethod(foo)
    foo2=classmethod(foo2)
        
>>> a = A()
>>> a.foo()
I am foo

>>> A.foo()
I am foo

>>> A.foo2()
I am foo2 <class '__main__.A'>

>>> a.foo3()
I am foo3 <__main__.A object at 0x00A1AA10>
Note that classes are callable objects by themselves and operate as instance factories. When you "call" a class you get an instance of that class as a result.

A different kind of callable object is an object that has a __call__ method. If you want to pass around a function-like object with its context intact, __call__ can be a good thing. Listing 1 features a simple 'add' function that can be replaced with a caching adder class that stores results of previous calculations. First, notice that the test function expects a function-like object called 'add' and it just invokes it as a function. The 'test' function is called twice-once with a simple function and a second time with the caching adder instance. Continuations in Python can also be implemented using __call__ but that's another article.

Metaclasses
Metaclasse is a concept that doesn't exist in today's mainstream programming languages. A metaclass is a class whose instances are classes. You already encountered a meta-class in this article called 'type'. When you invoke "type" with a class name, a base-classes tuple, and an attribute dictionary, the method creates a new user-defined class of the specified type. So the __class__ attribute of every class always contains its meta-class (normally 'type').

That's nice, but what can you do with a metaclass? It turns out, you can do plenty. Metaclasses allow you to control everything about the class that will be created: name, base classes, methods, and fields. How is it different from simply defining any class you want or even creating a class dynamically on the fly? Well, it allows you to intercept the creation of classes that are predefined as in aspect-oriented programming. This is a killer feature that I'll be discussing in a follow-up to this article.

After a class is defined, the interpreter looks for a meta-class. If it finds one it invokes its __init__ method with the class instance and the meta-class gets a stab at modifying it (or returning a completely different class). The interpreter will use the class object returned from the meta-class to create instances of this class.

So, how do you stick a custom metaclass on a class (new-style classes only)? Either you declare a __metaclass__ field or one of your ancestors has a __metaclass__ field. The inheritance method is intriguing because Python allows multiple inheritance. If you inherit from two classes that have custom metaclasses you are in for a treat-one of the metaclasses must derive from another. The actual metaclass of your class will be the most derived metaclass:


class M1(type): pass
class M2(M1):   pass

class C2(object): __metaclass__=M2    
class C1(object): __metaclass__=M1
class C3(C1, C2): pass


classes = [C1, C2, C3]
for c in classes:
    print c, c.__class__
    print '------------'                 

Output:
    
<class '__main__.C1'> <class '__main__.M1'>
------------
<class '__main__.C2'> <class '__main__.M2'>
------------
<class '__main__.C3'> <class '__main__.M2'>

Day In The Life of a Python Object
To get a feel for all the dynamics involved in using Python objects let's track a plain object (no tricks) starting from its class definition, through its class instantiation, access its attributes, and see it to its demise. Later on I'll introduce the hooks that allow you to control and modify this workflow.

The best way to go about it is with a monstrous simulation. Listing 2 contains a simulation of a bunch of monsters chasing and eating some poor person. There are three classes involved: a base Monster class, a MurderousHorror class that inherits from the Monster base class, and a Person class that gets to be the victim. I will concentrate on the MurderousHorror class and its instances.

Class Definition
MurderousHorror inherits the 'frighten' and 'eat' methods from Monster and adds a 'chase' method and a 'speed' field. The 'hungry_monsters' class field stores a list of all the hungry monsters and is always available through the class, base class, or instance (Monster.hungry_monsters, MurderousHorror.hungry_monsters, or m1.hungry_monsters). In the code below you can see (via the handy 'dir' function) the MurderousHorror class and its m1 instance. Note that methods such as 'eat,' 'frighten,' and 'chase' appear in both, but instance fields such as 'hungry' and 'speed' appear only in m1. The reason is that instance methods can be accessed through the class as unbound methods, but instance fields can be accessed only through an instance.


class NoInit(object):
    def foo(self):
        self.x = 5
        
    def bar(self):
        print self.x
                            
if __name__ == '__main__':            
    ni = NoInit()
    assert(not ni.__dict__.has_key('x'))
    try:
        ni.bar()
    except AttributeError, e:
        print e
    ni.foo()
    assert(ni.__dict__.has_key('x'))
    ni.bar()
    
Output:

'NoInit' object has no attribute 'x'
5
Object Instantiation and Initialization
Instantiation in Python is a two-phase process. First, __new__ is called with the class as a first argument, and later as the rest of the arguments, and should return an uninitialized instance of the class. Afterward, __init__ is called with the instance as first argument. (You can read more about __new__ in the Python reference manual.)

When a MurderousHorror is instantiated __init__ is the first method called. __init__ is similar to a constructor in C++/Java/C#. The instance calls the Monster base class's __init__ and initializes its speed field. The difference between Python and C++/Java/C# is that in Python there is no notion of a parameter-less default constructor, which, in other languages, is automatically generated for every class that doesn't have one. Also, there is no automatic call to the base class' default __init__ if the derived class doesn't call it explicitly. This is quite understandable since no default __init__ is generated.

In C++/Java/C# you declare instance variables in the class body. In Python you define them inside a method by explicitly specifying 'self.SomeAttribute'. So, if there is no __init__ method to a class it means its instances have no instance fields initially. That's right. It doesn't HAVE any instance fields. Not even uninitialized instance fields.

The previous code sample (above) is a perfect example of this phenomenon. The NoInit class has no __init__ method. The x field is created (put into its __dict__) only when foo() is called. When the program calls ni.bar() immediately after instantiation the 'x' attribute is not there yet, so I get an 'AttributeError' exception. Because my code is robust, fault tolerant, and self healing (in carefully staged toy programs), it bravely recovers and continues to the horizon by calling foo(), thus creating the 'x' attribute, and ni.bar() can print 5 successfully.

Note that in Python __init__ is not much more then a regular method. It is called indeed on instantiation, but you are free to call it again after initialization and you may call other __init__ methods on the same object from the original __init__. This last capability is also available in C#, where it is called constructor chaining. It is useful when you have multiple constructors that share common initialization, which is also one of the constructors/initializers. In this case you don't need to define another special method that contains the common code and call it from all the constructors/initializers; you can just call the shared constructor/initializer directly from all of them.

Attribute Access
An attribute is an object that can be accessed from its host using the dot notation. There is no difference at the attribute access level between methods and fields. Methods are first-class citizens in Python. When you invoke a method of an object, the method object is looked up first using the same mechanism as a non-callable field. Then the () operator is applied to the returned object. This example demonstrates this two-step process:


class A(object):
    def foo(self):
        print 3
                            
if __name__ == '__main__':            
    a = A()
    f = a.foo
    print f
    print f.im_self
    a.foo()
    f()
    
    
Output:

<bound method A.foo of <__main__.A object at 0x00A03EB0>>
<__main__.A object at 0x00A03EB0>
3
3
The code retrieves the a.foo bound method object and assigns it to a local variable 'f'. 'f' is a bound method object, which means its im_self attribute points to the instance to which it is bound. Finally, a.foo is invoked through the instance (a.foo()) and by calling f directly with identical results. Assigning bound methods to local variables is a well known optimization technique due to the high cost of attribute lookup. If you have a piece of Python code that seems to perform under the weather there is a good chance you can find a tight loop that does a lot of redundant lookups. I will talk later about all the ways you can customize the attribute access process and why it is so costly.

Destruction
The __del__ method is called when an instance is about to be destroyed (its reference count reaches 0). It is not guaranteed that the method will ever be called in situations such as circular references between objects or references to the object in an exception. Also the implementation of __del__ may create a new reference to its instance so it will not be destroyed after all. Even when everything is simple and __del__ is called, there is no telling when it will actually be called due to the nature of the garbage collector. The bottom line is if you need to free some scarce resource attached to an object do it explicitly when you are done using it and don't wait for __del__.

A try-finally block is a popular choice for garbage collection since it guarantees the resource will be released even in the face of exceptions. The last reason not to use is __del__ is that its interaction with the 'del' built-in function may confuse programmers. 'del' simply decrements the reference count by 1 and doesn't call '__del__' or cause the object to be magically destroyed. In the next code sample I use the sys.getrefcount() function to determine the reference count to an object before and after calling 'del'. Note that I subtract 1 from sys.getrefcount() result because it also counts the temporary reference to its own argument.


import sys

class A(object):
    def __del__(self):
        print "That's it for me"
                            
if __name__ == '__main__':            
    a = A()
    b = a
    print sys.getrefcount(a)-1
    del b
    print sys.getrefcount(a)-1

Output:

2
1
That's it for me


Hacking Python

Let the games begin. In this section I will explore different ways to customize attribute access. The topics include the __getattribute__ hook, descriptors, and properties.


- __getattr__, __setattr__ and __getattribute__
These special methods control attribute access to class instances. The standard algorithm for attribute lookup returns an attribute from the instance dictionary or one of its base class's dictionaries (descriptors will be described in the next section). They are supposed to return an attribute object or raise AttributeError exception. If you define some of these methods in your class they will be called upon during attribute access under some conditions. Listing 3 is an interactive example. It is designed to allow you to play around with it and comment out various functions to see the effect. It introduces the class A with a single 'x' attribute. It has __getattr__, __setattr__, and __getattribute__ methods. __getattribute__ and __setattr__ simply forward any attribute access to the default (lookup or set value in dictionary). __getattr__ always returns 7. The main program starts by assigning 6 to the non-existing attribute 'y' (happens via __setattr__) and then prints the preexisting 'x', the newly created 'y', and the still non-existent 'z'. 'x' and 'y' exist now, so they are accessible via __getattribute__. 'z' doesn't exist so __getattribute__ fails and __getattr__ gets called and returns 7. (Author's Note: This is contrary to the documentation. The documentation claims if __getattribute__ is defined, __getattr__ will never be called, but this is not the actual behavior.)

Descriptors
A descriptor is an object that implements three methods __get__, __set__, and __delete__. If you put such a descriptor in the __dict__ of some object then whenever the attribute with the name of the descriptor is accessed one of the special methods is executed according to the access type (__get__ for read, __set__ for write, and __delete__ for delete).This simple enough indirection scheme allows total control on attribute access.

The following code sample shows a silly write-only descriptor used to store passwords. Its value may not be read nor deleted (it throws AttributeError exception). Of course the descriptor object itself and the password can be accessed directly through A.__dict__['password'].


class WriteOnlyDescriptor(object):
    def __init__(self):
        self.store = {}

    def __get__(self, obj, objtype=None):
        raise AttributeError 

    def __set__(self, obj, val):
        self.store[obj] = val
    
    def __del(self, obj):
        raise AttributeError

class A(object):
    password = WriteOnlyDescriptor()
      
if __name__ == '__main__': 
    a = A()
    try:
        print a.password
    except AttributeError, e:
        print e.__doc__
    a.password = 'secret'
    print A.__dict__['password'].store[a]
Descriptors with both __get__ and __set__ methods are called data descriptors. In general, data descriptors take lookup precedence over instance dictionaries, which take precedence over non-data descriptors. If you try to assign a value to a non-data descriptor attribute the new value will simply replace the descriptor. However, if you try to assign a value to a data descriptor the __set__ method of the descriptor will be called.

Properties
Properties are managed attributes. When you define a property you can provide get, set, and del functions as well as a doc string. When the attribute is accessed the corresponding functions are called. This sounds a lot like descriptors and indeed it is mostly a syntactic sugar for a common case.

This final code sample is another version of the silly password store using properties. The __password field is "private." Class A has a 'password' property that, when accessed as in 'a.password,' invokes the getPassword or setPassword methods. Because the getPassword method raises the AttributeError exception, the only way to get to the actual value of the __password attribute is by circumventing the Python fake privacy mechanism. This is done by prefixing the attribute name with an underscore and the class name a._A__password. How is it different from descriptors? It is less powerful and flexible but more pleasing to the eye. You must define an external descriptor class with descriptors. This means you can use the same descriptor for different classes and also that you can replace regular attributes with descriptors at runtime.


class A(object):
    def __init__(self):
        self.__password = None

    def getPassword(self):
        raise AttributeError

    def setPassword(self, password):        
        self.__password = password

    password = property(getPassword, setPassword)    
      
if __name__ == '__main__':
    a = A()
    try:
        print a.password
    except AttributeError, e:
        print e.__doc__
    a.password = 'secret'
    print a._A__password
    
Output:
    
Attribute not found.
secret 
Properties are more cohesive. The get, set functions are usually methods of the same class that contain the property definition. For programmers coming from languages such as C# or Delphi, Properties will make them feel right at home (too bad Java is still sticking to its verbose java beans).

Python's Richness a Mixed Blessing
There are many mechanisms to control attribute access at runtime starting with just dynamic replacement of attribute in the __dict__ at runtime. Other methods include the __getattr__/__setattr, descriptors, and finally properties. This richness is a mixed blessing. It gives you a lot of choice, which is good because you can choose whatever is appropriate to your case. But, it is also bad because you HAVE to choose even if you just choose to ignore it. The assumption, for better or worse, is that people who work at this level should be able to handle the mental load.

In my next article, I will pick up where I've left off. I'll begin by contrasting metaclasses with decorators, then explore the Python execution model, and explain how to examine stack frames at runtime. Finally, I'll demonstrate how to augment the Python language itself using these techniques. I'll introduce a private access checking feature that can be enforced at runtime.

Gigi Sayfan is a software developer working on CELL applications for Sony Playstation3. He specializes in cross-platform object-oriented programming in C/C++/C#/Python with emphasis on large-scale distributed systems.

[Feb 20, 2006] freshmeat.net Project details for Meld

Meld is a GNOME 2 visual diff and merge tool. It integrates especially well with CVS. The diff viewer lets you edit files in place (diffs update dynamically), and a middle column shows detailed changes and allows merges. The margins show location of changes for easy browsing, and it also features a tabbed interface that allows you to open many diffs at once.

Information about CWM - TimBL's Closed World Machine

CWM is a popular Semantic Web program that can do the following tasks:-

CWM was written in Python from 2000-10 onwards by Tim Berners-Lee and Dan Connolly of the W3C.

This resource is provided so that people can use CWM, find out what it does (documentation used to be sparse), and perhaps even contribute to its development.

What's new in Python 2.4

New or upgraded built-ins

Extending and Embedding the Python Interpreter

Extending Python with C (Score:1)
by frehe (6916) <frehe491@NoSpaM.student.liu.se> on Wednesday April 16, @03:41PM (#5745981)

If you need more speed than native Python provides, you can always write code in C and wrap it so it is callable from Python. The wrapping is really easy to do, once you have understood the general concepts involved in it. The product I currently work on has about 10000 lines of C code (crypto and networking) which is used this way, and it works perfectly. For more information about extending Python with C, see:

Extending and Embedding the Python Interpreter [python.org]

Dive Into Python Python for experienced programmers

Dive Into Python is a free Python book for experienced programmers. You can read the book online, or download it in a variety of formats. It is also available in multiple languages.

This book is still being written. The first three chapters are a solid overview of Python programming. Chapters covering HTML processing, XML processing, and unit testing are complete, and a chapter covering regression testing is in progress. This is not a teaser site for some larger work for sale; all new content will be published here, for free, as soon as it's ready. You can read the revision history to see what's new. Updated 28 July 2002

Wing IDE for Python Python IDS that includes source browser and editor. The editor supports folding

Reference Manual Wing IDE Version 1.1.4 Pythod IDS that includes source broser and editor.

developerWorks Linux Open source projects Charming Python Iterators and simple generators

What's New in Python 2.2 -- generators is a very interesting feature of Python 2.2 that is essentially a co-routine.

Generators are another new feature, one that interacts with the introduction of iterators.

You're doubtless familiar with how function calls work in Python or C. When you call a function, it gets a private namespace where its local variables are created. When the function reaches a return statement, the local variables are destroyed and the resulting value is returned to the caller. A later call to the same function will get a fresh new set of local variables. But, what if the local variables weren't thrown away on exiting a function? What if you could later resume the function where it left off? This is what generators provide; they can be thought of as resumable functions.

Here's the simplest example of a generator function:

def generate_ints(N):
    for i in range(N):
        yield i

A new keyword, yield, was introduced for generators. Any function containing a yield statement is a generator function; this is detected by Python's bytecode compiler which compiles the function specially as a result. Because a new keyword was introduced, generators must be explicitly enabled in a module by including a from __future__ import generators statement near the top of the module's source code. In Python 2.3 this statement will become unnecessary.

When you call a generator function, it doesn't return a single value; instead it returns a generator object that supports the iterator protocol. On executing the yield statement, the generator outputs the value of i, similar to a return statement. The big difference between yield and a return statement is that on reaching a yield the generator's state of execution is suspended and local variables are preserved. On the next call to the generator's .next() method, the function will resume executing immediately after the yield statement. (For complicated reasons, the yield statement isn't allowed inside the try block of a try...finally statement; read PEP 255 for a full explanation of the interaction between yield and exceptions.)

Here's a sample usage of the generate_ints generator:

>>> gen = generate_ints(3)
>>> gen
<generator object at 0x8117f90>
>>> gen.next()
0
>>> gen.next()
1
>>> gen.next()
2
>>> gen.next()
Traceback (most recent call last):
  File "<stdin>", line 1, in ?
  File "<stdin>", line 2, in generate_ints
StopIteration

You could equally write for i in generate_ints(5), or a,b,c = generate_ints(3).

Inside a generator function, the return statement can only be used without a value, and signals the end of the procession of values; afterwards the generator cannot return any further values. return with a value, such as return 5, is a syntax error inside a generator function. The end of the generator's results can also be indicated by raising StopIteration manually, or by just letting the flow of execution fall off the bottom of the function.

You could achieve the effect of generators manually by writing your own class and storing all the local variables of the generator as instance variables. For example, returning a list of integers could be done by setting self.count to 0, and having the next() method increment self.count and return it. However, for a moderately complicated generator, writing a corresponding class would be much messier. Lib/test/test_generators.py contains a number of more interesting examples. The simplest one implements an in-order traversal of a tree using generators recursively.

# A recursive generator that generates Tree leaves in in-order.
def inorder(t):
    if t:
        for x in inorder(t.left):
            yield x
        yield t.label
        for x in inorder(t.right):
            yield x

Two other examples in Lib/test/test_generators.py produce solutions for the N-Queens problem (placing queens on an chess board so that no queen threatens another) and the Knight's Tour (a route that takes a knight to every square of an chessboard without visiting any square twice).

The idea of generators comes from other programming languages, especially Icon (http://www.cs.arizona.edu/icon/), where the idea of generators is central. In Icon, every expression and function call behaves like a generator. One example from ``An Overview of the Icon Programming Language'' at http://www.cs.arizona.edu/icon/docs/ipd266.htm gives an idea of what this looks like:

sentence := "Store it in the neighboring harbor"
if (i := find("or", sentence)) > 5 then write(i)

In Icon the find() function returns the indexes at which the substring ``or'' is found: 3, 23, 33. In the if statement, i is first assigned a value of 3, but 3 is less than 5, so the comparison fails, and Icon retries it with the second value of 23. 23 is greater than 5, so the comparison now succeeds, and the code prints the value 23 to the screen.

Python doesn't go nearly as far as Icon in adopting generators as a central concept. Generators are considered a new part of the core Python language, but learning or using them isn't compulsory; if they don't solve any problems that you have, feel free to ignore them. One novel feature of Python's interface as compared to Icon's is that a generator's state is represented as a concrete object (the iterator) that can be passed around to other functions or stored in a data structure.

See Also:

PEP 255, Simple Generators
Written by Neil Schemenauer, Tim Peters, Magnus Lie Hetland. Implemented mostly by Neil Schemenauer and Tim Peters, with other fixes from the Python Labs crew.

Dive Into Python Dive Into Python is a free Python book for experienced programmers.

You can read the book online, or download it in a variety of formats. It is also available in multiple languages.

Seventh International Python Conference Papers

Applications I: The Internet

Optimizing Python

Extending and Compiling

Applications II: Science and Simulation

Slashdot: What Makes a Powerful Programming Language

by NeuroMorphus on Monday February 11, @07:26PM (#2991161)
(User #463324 Info | http://slashdot.org/)
You also have my Python vote. Python is not only a powerful language on its own, but it's also a good glue language, pulling C++ together to speed up particular modules, or using Jython for a Java perspective(IMHO, I like programming in Jython better than Java itself:)

Python is also good for server-side scripting as well as operating system scripts. Not to mention that Python has lots of support including, Twisted Matrix [twistedmatrix.com], which is an event-based framework for internet applications that include a web server, a telnet server, a multiplayer RPG engine, a generic client and server for remote oject access, and API's for creating new protocols and services.

If you want GUI design, you have wxWindows, wxPython for wxGTK, Tk, PyQt, PyGTK, etc.

I would tell your boss to got with Python; You'll end up using it down the road anyway ;)

Consider Python carefully (Score:3, Informative)
by pclminion on Monday February 11, @09:17PM (#2991850)
(User #145572 Info)
I can definitely speak for the elegance of Python; I've used it to build some pretty large-scale AI projects, among about a zillion other things.

If you're looking for the weirder OO features like operator overloading, you'll find them in Python, but the calisthenics you have to go through to do it might make you think twice about using it.

The only real drawback to Python is the execution speed. One of the AI projects I previously mentioned worked great (and I completed it probably 10 times faster than I would have had I been using C/C++), but it ran very very slowly. The cause of the problem is clear -- no static typing, reference-count garbage collection, a stack-based VM (I ain't knockin' it! It's just hard to optimize)...

If you are planning to do anything compute-intensive, maybe Python is not the right choice. It's always possible to break out to C/C++ with the Python API (to do your compute-intensive tasks), but there are drawbacks: over time, the Python C-API will probably drift, and you'll have to keep tweaking your native code to keep it working. Also, the inner workings of Python, especially refcounts, can be boggling and the source of bugs and memory leaks that can be fantastically hard to track down.

If you consider Python, then great. Just keep these points in mind.

Python for you and "Clean C" for Cliff (Score:1)
by Nice2Cats on Tuesday February 12, @04:56AM (#2993069)
(User #557310 Info | http://slashdot.org/)
It does sound like your boss was trying to tell you to use Python, unless it needs to be blindingly fast (as in "operating system" or "3d shooter"). I'd like to second that recommendation and the arguments already posted.

To respond to Cliff's question: The one thing that every new language should have is that "significant whitespace" look-ma-no-braces syntax used by Python - or at least the option of not having to spend your life typing semicolons.

In fact, you could probably retrofit the common C and C++ compilers to accept significant whitespace. This would mean the Linux kernel people would be reading:

static int proc_sel(struct task_struct *p, int which, int who):
if(p->pid):
switch (which):
case PRIO_PROCESS:
if (!who && p == current):
return 1
return(p->pid == who)
case PRIO_PGRP:
if (!who):
who = current->pgrp
return(p->pgrp == who)
case PRIO_USER:
if (!who):
who = current->uid
return(p->uid == who)
return 0

(Modified from /linux/kernel/sys.c (C) Linus Torvalds)

Of course, there is a problem of finding a new name for this sort of C without violating somebodys copyright...is "Clean C" still available?

[Jun 12, 2003] Text Processing in Python by David Mertz

Notable quotes:
"... What is truly great about the book is the discussion of state machines, parsers, and functional programming. Although these topics detract from the focus on string processing somewhat this book is perhaps the only popular Python book out there that does these topics justice. I thought they were very well written. ..."
"... I found string processing to be messy with Python but found Ruby to be much easier. That is perhaps because Ruby is a newer language and it has some features of Perl built in. Ruby however does not have the extent of libraries available like Python, nor does it have as nice of Windows GUI. ..."
www.amazon.com

John C. Dunbar on February 28, 2004

Round the world tour of string processing for Pythoneers

This is the only book that really attacks the issue of string processing using Python. Unfortunately it didn't attack the text processing problems that I wanted discussed.

Also, in the area of Regular Expressions the examples didn't directly use the Python library, instead a wrap around function was used for the many examples and that detracted from using the book as a reference book for this purpose.

I found that Python has several different ways to do string processing. Also, some of those ways come up with conflicting results. At the time of this writing the authors of Python are re-organizing and improving this area.

What is truly great about the book is the discussion of state machines, parsers, and functional programming. Although these topics detract from the focus on string processing somewhat this book is perhaps the only popular Python book out there that does these topics justice. I thought they were very well written.

My overall complaint is that this book includes too many things outside of text processing using the core Python language. But other readers may appreciate this aspect more than I did. If you want coverage on handling email specifically, the author covers that. Same with HTML processing and other specialized topics. I just wanted to low down on using the full string processing capabilities of the core Python language -- not necessarily all the specialized libraries.

I found string processing to be messy with Python but found Ruby to be much easier. That is perhaps because Ruby is a newer language and it has some features of Perl built in. Ruby however does not have the extent of libraries available like Python, nor does it have as nice of Windows GUI.

Overall, if you are looking for a book on text processing this is the only book out there, and a big plus with this book is what you will learn on function programming, state machines and parsers.

The author worked hard to produce a book in this specialized area. He has lots of code examples. Highly recommended for Python programmers.

John Dunbar
Sugar Land, TX

[Dec 28, 2001] http://www.dotfunk.com/projects/pylint/pylint.0.1.tar.gz

Pylint is a static type checker for Python (compare with PyChecker 0.4) David Jeske and Scott Hassan proved that it is possible to do this. They are working on a type inference engine that understands the Python language and can detect type errors and violations. Adding type checking to python without
changing the language will ease the maintenance of a large python project with lots of developers. Please remember that eGroups before it was bought by Yahoo was a huge Python project (more than 180,000 lines of Python doing everything from a 100% dynamic website to all email delivery, pumping out 200 messages/second on a single 400 MHz Pentium!)

[Dec 20, 2001] Semi-coroutines in Python 2.2

Search Result 31

From: Steven Majewski (sdm7g@Virginia.EDU)
Subject: Re: (semi) stackless python
Newsgroups: comp.lang.python

View: Complete Thread (2 articles) | Original Format

Date: 2001-12-20 13:01:46 PST

 The new generators in Python2.2 implement semi-coroutines, not
full coroutines:  the limitation is that they always return to
their caller -- they can't take an arbitrary continuation as
a return target.

 So you can't do everything you can do in Stackless, or at least,
you can't do it the same way. I'm not sure what the limitations
are yet, but you might be surprised with what you CAN do.
 If all the generator objects yield back to the same 'driver'
procedure, then you can do a sort of cooperative multithreading.
In that case, you're executing the generators for their side
effects -- the value returned by yield may be unimportant except
perhaps as a status code.

[ See also Tim Peters' post on the performance advantages of
  generators -- you only parse args once for many generator
  calls, and you keep and reuse the same stack frame. So I
  believe you get some of the same benefits of Stackless
  microthreads. ]

Maybe I can find the time to post an example of what I mean.
In the mean time, some previous posts on generators might give
you some ideas.

<http://groups.google.com/groups?q=generators+group:comp.lang.python+author:Majewski&hl=en&scoring=d&as_drrb=b&as_mind=12&as_minm=1&as_miny=2001&as_maxd=20&as_maxm=12&as_maxy=2001&rnum=1&selm=mailman.1008090197.6318.python-list%40python.org>

<http://groups.google.com/groups?hl=en&threadm=mailman.996870501.28562.python-list%40python.org&rnum=6&prev=/groups%3Fas_q%3Dgenerators%26as_ugroup%3Dcomp.lang.python%26as_uauthors%3DMajewski%26as_drrb%3Db%26as_mind%3D12%26as_minm%3D1%26as_miny%3D2001%26as_maxd%3D20%26as_maxm%3D12%26as_maxy%3D2001%26num%3D100%26as_scoring%3Dd%26hl%3Den>

The ungrammatical title of that last thread above:
"Nested generators is the Python equivalent of unix pipe cmds."
might also suggest the solution I'm thinking of.


-- Steve Majewski

The Python Refactoring Browser

Jun 06, 2000, 22:08 UTC ) ( Published ) ( By Shae Erisson ) ( Seen at comp.lang.python ) ( Submit by Geoff )

[ Shae Erisson writes: Help write the Refactoring Browser for Python :) I think it would improve many things about medium to large Python projects. https://sourceforge.net/project/?group_id=4298 ]

The Python Refactoring Browser, helping Pythonistas everywhere glide over the gory details of refactoring their code. Watch him extract jumbled code into well ordered classes. Gasp, as he renames all occurrences of a method.

Thank You Bicycle Repair Man!

TechNetCast Archives

PyChecker 0.4 PyChecker is a python source code checking tool to help you find common bugs.

[Vaults of Parnassus]

Python for Lisp Programmers

This is a brief introduction to Python for Lisp programmers. Basically, Python can be seen as a dialect of Lisp with "traditional" syntax (what Lisp people call "infix" or "m-lisp" syntax). One message on comp.lang.python said "I never understood why LISP was a good idea until I started playing with python." Python supports all of Lisp's essential features except macros, and you don't miss macros all that much because it does have eval, and operator overloading, so you can create custom languages that way. (Although it wasn't my intent, Python programmers have told me this page has helped them learn Lisp.)

I looked into Python because I was considering translating the code for the Russell & Norvig AI textbook from Lisp to Java so that I could have (1) portable GUI demos, (2) portable http/ftp/html libraries, (3) free development environments on all major platforms, and (4) a syntax for students and professors who are afraid of parentheses. But writing all that Java seemed much too daunting, and I think that students who used Java would suffer by not having access to an interactive environment to try things out. Then I discovered JPython, a version of Python that is neatly integrated into Java, giving us access to the Java GUIs. Of course, Python already has web libraries, so JPython can use either those or Java's.

[Nov 02, 2001] Perl to Python Migration by Martin C. Brown

Martin C Brown is the author of several books on Perl (including Debugging Perl) and a book on Python (Python The Complete Reference, 2001) so he knows both languages. which makes the book pretty unique. there is no other books that cover Python from the position of Perl programmer.
My conclusion

Python is an excellent language for my intended use. It is a good language for many of the applications that one would use Lisp as a rapid prototyping environment for. The three main drawbacks are (1) execution time is slow, (2) there is very little compile-time error analysis, even less than Lisp, and (3) Python isn't called "Java", which is a requirement in its own right for some of my audience. I need to determine if JPython is close enough for them.

Python can be seen as either a practical (better libraries) version of Scheme, or as a cleaned-up (no $@&%) version of Perl. While Perl's philosophy is TIMTOWTDI (there's more than one way to do it), Python tries to provide a minimal subset that people will tend to use in the same way. One of Python's controversial features, using indentation level rather than begin/end or {/}, was driven by this philosophy: since there are no braces, there are no style wars over where to put the braces. Interestingly, Lisp has exactly the same philosophy on this point: everyone uses emacs to indent their code. If you deleted the parens on control structure special forms, Lisp and Python programs would look quite similar.

Python has the philosophy of making sensible compromises that make the easy things very easy, and don't preclude too many hard things. In my opinion it does a very good job. The easy things are easy, the harder things are progressively harder, and you tend not to notice the inconsistencies. Lisp has the philosophy of making fewer compromises: of providing a very powerful and totally consistent core. This can make Lisp harder to learn because you operate at a higher level of abstraction right from the start and because you need to understand what you're doing, rather than just relying on what feels or looks nice. But it also means that in Lisp it is easier to add levels of abstraction and complexity; Lisp makes the very hard things not too hard.

[May 6, 2001] Zope for the Perl-CGI programmer by Michael Roberts (michael@vivtek.com) Zope is written in Python

Zope (the Z Object Publishing Environment) is an application server that is gaining in popularity. But what is it? What's an application server, anyway? How does all this compare with nice familiar paradigms like CGI? More importantly, is Zope a fad, or is it here to stay?

In a nutshell, here's what you get from Zope:

And you get all this in an easily installed, easily maintained package. But the most significantly new thing about Zope is how it encourages you to look at your Web site differently than you do now. Let's talk about that a little before we get down to brass tacks.

Python creator: Perl users are moving to Python

searchEnterpriseLinux: Are there situations in which it's better to use Perl than Python, or vice versa?
van Rossum: They're both competing for the same niche, in some cases. If all you need to do is simple text processing, you might use Perl. Python, much more than Perl, encourages clean coding habits to make it easy for other people to follow what you are doing. I've seen a lot of people who were developing in Perl moving to Python because it's easier to use. Where Python wins is when you have users who are not very sophisticated but have to write some code. An example would be in educational settings, where you have students who have no prior programming experience. Also, Python seems to work better when you have to write a large system, working together in a large group of developers, and when you want the lifetime of your program to be long.
searchEnterpriseLinux: In what situations would people use Java instead of Python?
van Rossum: Java and Python have quite different characteristics. Java is sophisticated language. It takes a longer time to learn Java than Python. With Java, you have to get used to the compile-edit-run cycle of software development. Python is a lighter-weight language. There is less to learn to get started. You can hit the ground running when you have only a few days of Python training under your belt. Python and Java can be used in the same project. Python would be used for higher level control of an application, and Java would be used for implementing the lower level that needs to run relatively efficiently. Another place where Python is used is an extension language. An example of that is Object Domain, a UML editing tool written in Java. It uses Python as an extension language.
searchEnterpriseLinux: Will use of Python in developing applications increase?
van Rossum: It's going to increase dramatically. There's a definite need for a language that's as easy to use as Python. Python is used to teach students how to program, so that is creating a base of people who know and like the language.
searchEnterpriseLinux: Are there technology developments that will benefit Linux developers?
van Rossum: Python developer tools are becoming available that work well in the Linux world. Until a year ago, the Python developer was stuck with using Emacs or VI to edit source code. Both of those editors have support for Python, but they're just editors. Now there are tools like Wing IDE, which is a development tool that is written in Python. Komodo is more ambitious and less finished at the moment, and it also supports program development with Perl. There is a Swedish company called Secret Labs which have had Windows development tools for Python, but which is now focusing on the Linux market. As more development environments become available for Python, we'll see a lot more users on the Linux platform.

Dave Warner wrote an excellent article on XML-RPC and Python here, using Meerkat as an example of a real XML-RPC server

Another article about XML-RPC

According to Jeff Walsh's InfoWorld article, Microsoft is planning to open up their operating system using XML-RPC. Such a protocol could be deployed quickly in other operating systems that support HTTP, ranging from Perl scripts running on Linux, to Quark publishing systems running on Macs, to relational databases running on mainframe systems. It could put Windows at the center of a new kind of web, one built of logic, storing objects and methods, not just pages and graphics.

The XML-RPC HOWTO is here.

People interested in XML-RPC might be interested in checking it out using KDE. Since KDE 2.0 a DCOP XMLRPC bridge has been included allowing easy access to a wide range of the desktops APIs.

My experiencie implementing XML-RPC (Score:3, Interesting)
by Nicopa (nick@debian.org) on Tuesday February 20, @08:04PM EST (#88)
(User #87617 Info)
This is my experience with XML-RPC:

I work for a company (Technisys) which have created several years ago an RPC tool called "tmgen". This tool is built as a layer on top of rpcgen, adding session cookie handling, SSL support, a stateless server, handling of enumerated values with associated long and short descriptions, and many other thing. It's in fact, an application server built on top of RPC.

This baby have been running for many years in the most important banks and credit cards companies here in Argentina (yes, you know the brands, but I'm not sure I can tell you which ones =) ).

The "tmgen" tool reads a ".def" file that defines the datatyes, and ".trn" files which have the code of the different "transactions". Having read those files, it automatically generates the server source (including the rpcgen input source).

I was asked to make it possible for the clients to be programmed in the Java language. I evaluated several possibilities, one of them using a Java client for RPC. This required us to go for a proprietary solution, besides, being in control of both sides it looked silly to be tied to a protocol. Another possibility would have been to modify tmgen to create an RMI server. But the best was to create an XML server (IMO). I then evaluated SOAP and XML-RPC. SOAP seemed very nice, but XML-RPC was *direct* mapping of the semantics and types of our existing solution. The benefits of SOAP were a drawback in this case, we just wanted to have strings, structs, ints and floats.

So, now it's working. It takes a definition of the structs, the method and which parameters they get, and it creates code (using the gnome-xml library (which I recommend). The automatically generated code works as a standalone inetd webserver which reads an XML-RPC query from the net, parses the query, loads it in the generated C structures, run the "transaction", and creates a response from the reply structures. The final result was that all those old C, RPC-only, programs started to have an XML interface.

I added the Helma RPC-XML client and voila, we had a Java client. So I must say that my experience in this legacy system with XML-RPC was great.

Talking about new systems, I think that XML-RPC does the wrong thing, by defining markup for the types instead of marking the semantic of the data.

XML-RPC, SOAP, and the Rest... (Score:2, Informative)
by edashofy on Tuesday February 20, @08:17PM EST (#95)
(User #265252 Info)
I've used XML-RPC before, written my own version of it, and used SOAP extensively (and even posted a few patches to the IBM SOAP for Java implementation). My conclusion has been that while RPC is slow, XML-RPC (and its variants) are necessarily slower. The idea itself is good, and it may be useful for so-called "web services" where there is a strict client-server relationship between communicating machines and there are few RPC's bouncing around the network. However, the overhead of a full XML parser, plus a protocol implementation, plus a marshaller and demarshaller to and from XML (especially since no language really provides this) is a big problem for XML-RPC and its kin. According to Dave Winer... "Conceptually, there's no difference between a local procedure call and a remote one, but they are implemented differently, perform differently (RPC is much slower) and therefore are used for different things." Bzzt! A local procedure call can pass references, pointers, etc. A remote procedure call is not just slower, but it also limits what kind of data you can reasonably expect to pass in the parameters. A pointer to a database driver, for instance, or a function pointer (for callbacks) are entirely different beasts! From the slashdot header: "It's deliberately minimalist but nevertheless quite powerful, offering a way for the vast majority of RPC applications that can get by on passing around boolean/integer/float/string datatypes to do their thing in a way that is lightweight and easy to to understand and monitor" Lightweight? Simple to understand and monitor? First of all, XML parsers are NEVER lightweight, especially if you want to do SOAP and use things like namespaces. Second of all, if you're doing only booleans/integers/floats/strings as the above suggests, then you're fine. More complex datatype marshalling? Ouch! I'm going to leave XML-RPC, SOAP, and its kin behind for a while and see what happens. Until then, caveat earlius adopterus!
Why isn't XML-RPC considered bloat? (Score:3, Interesting)
by Carnage4Life on Tuesday February 20, @09:16PM EST (#116)
(User #106069 Info) http://www.25hoursaday.com
I am a big fan of distributed computing, heck I wrote an article about it on K5, and have always wondered what the XML-RPC payoff is.

From what I can tell, XML-RPC is a way to replace the binary protocols that current distributed systems use (e.g. Corba's IIOP or DCOM's ORPC) with an XML based one. So when an object needs perform a remote method call, instead of just sending it's arguments in a compact efficient binary packet, it builds an XML string which has to be parsed on the other end. So with XML RPC, remote method calls now need the addition of an XML parser to their current bag of tricks.

On the surface it seems that this makes it easier to perform distributed computing since any shmuck can use an XML parser and call a few functions. But it means that an extra layer of abstraction has been added to operations that should be performed rather quickly for the dubious benefit of compatibility across platforms (which is yet to be realized) which seems to be more jumping on the XML hype bandwagon than reality. My biggest issue is that for XML-RPC to support things that are the biggest issues of distributed computing (e.g. keeping track of state) would add so much bloat to the XML parsing, string building, etc process for making a remote call as to make it unfeasible.

Anyone see any errors in this thinking?

Grabel's Law
2 is not equal to 3 - not even for very large values of 2.

Re:Why isn't XML-RPC considered bloat? (Score:4, Informative)
by Hrunting (hrunting@nospam.texas.net) on Tuesday February 20, @10:19PM EST (#134)
(User #2191 Info) http://hrunting.home.texas.net/
Anyone see any errors in this thinking?

Yes. It's true that as a consequence of its text-based nature, communication via XML is decidedly more bandwidth-intensive than any binary counterpart. The problem, though, lies in translation of that binary format two and from different machines, architectures, languages, and implementations.

I currently do quite a bit of work using SOAP, which is similar to XML-RPC, but a little less well-developed. It's a no-brainer. If I'm using Java, Perl, C, C++ or even Python, it's relatively easy to make calls between these different languages. I don't have to worry about the endianess of the data I'm working with. I don't have to worry about learning some new data encoding scheme (XML is very well-defined, and very ubiquitous; almost every language has a translator). Communicating between a language like Perl, which has no typing, and Java, which has more strict typing is a no brainer, because the data structures are defined by a well-documented, human readable schema, and when I look at the data I'm sending, I can see the raw information.

Bandwidth concerns might have been paramount two years ago, but the world in which XML-RPC and SOAP are being used has already shifted to broadband. Now, human clarity and complete interoperability, as well as the ease of use of porting XML-RPC (or SOAP) constructs to another language (since it's just XML text) make it a much more efficient model in terms of programmer time.

Yes, from a strictly bandwidth concern, CORBA or DCOM beat XML hands down, but when you remove that consideration, it's not that big a deal. Couple it with existing protocols (I've seen SOAP implementations via HTTP, FTP, and even SMTP), and the opportunity to grow via existing infrastructure and well-known technologies, and you just have an easier (and thus, I would argue, more open) model to work with.

Random links

O'Reilly Network: Introduction to Stackless Python(Oct 19, 2000)

Linux.com: Programming with Python - Part 3: Extending Python(Nov 19, 2000)

Security Portal: Python: Security Aspects (Nov 17, 2000)

Python-dev summary, November 1-15, 2000 (Nov 16, 2000)

IBM developerWorks: Charming Python: Inside Python's implementations(Nov 13, 2000)

O'Reilly Network: Python Roadmap(Oct 22, 2000)

O'Reilly Network: Python 2.0 and Beyond(Oct 22, 2000

[Nov 22, 2000] LinuxPR: PythonWorks Pro v1.1 - Now Also for Linux and Solaris!

"PythonWorks Pro, entirely developed in Python, contains an editor, project manager, a deployment tool, an integrated browser, debugger, and a proprietary interface design tool. PythonWorks Pro allows the developer to easily create Python applications using the built-in tools."

[Nov 14, 2000] Linux Charming Python -- Inside Python's implementations See also Slashdot Interviews With The Creators of Vyper and Stackless

developerWorks

Stackless Python: An interview with creator Christian Tismer
At first brush, Stackless Python might seem like a minor fork to CPython. In terms of coding, Stackless makes just a few changes to the actual Python C code (and redefines "truth"). The concept that Christian Tismer (the creator of Stackless Python) introduces with Stackless is quite profound, however. It is the concept of "continuations" (and a way to program them in Python).

To attempt to explain it in the simplest terms, a continuation is a representation, at a particular point in a program, of everything the program is capable of doing subsequently. A continuation is a potential that depends on initial conditions. Rather than loop in a traditional way, it is possible to invoke the same continuation recursively with different initial conditions. One broad claim I have read is that continuations, in a theoretical sense, are more fundamental and underlie every other control structure. Don't worry if these ideas cause your brain to melt; that is a normal reaction.

Reading Tismer's background article in the Resources is a good start for further understanding. Pursuing his references is a good way to continue from there. But for now, let's talk with Tismer at a more general level:

Mertz: Exactly what is Stackless Python? Is there something a beginner can get his or her mind around that explains what is different about Stackless?

Tismer: Stackless Python is a Python implementation that does not save state on the C stack. It does have stacks -- as many as you want -- but these are Python stacks.

The C stack cannot be modified in a clean way from a language like C, unless you do it in the expected order. It imposes a big obligation on you: You will come back, exactly here, exactly in the reverse way as you went off.

"Normal" programmers do not see this as a restriction in the first place. They have to learn to push their minds onto stacks from the outset. There is nothing bad about stacks, and usually their imposed execution order is the way to go, but that does not mean that we have to wait for one such stack sequence to complete before we can run a different one.

Programmers realize this when they have to do non-blocking calls and callbacks. Suddenly the stack is in the way, we must use threads, or explicitly store state in objects, or build explicit, switchable stacks, and so on. The aim of Stackless is to deliver the programmer from these problems.

Mertz: The goal of Stackless is to be 100% binary compatible with CPython. Is it?

Tismer: Stackless is 100% binary compatible at the moment. That means: You install Python 1.5.2, you replace Python15.dll with mine, and everything still works, including every extension module. It is not a goal, it was a demand, since I didn't want to take care about all the extensions.

Mertz: Stackless Python has been absolutely fascinating to read about for me. Like most earthbound programmers, I have trouble getting my mind wholly around it, but that is part of what makes it so interesting.

Tismer: Well, I'm earthbound, too, and you might imagine how difficult it was to implement such a thing, without any idea what a continuation is and what it should look like in Python. Getting myself into doing something that I wasn't able to think was my big challenge. After it's done, it is easy to think, also to redesign. But of those six months of full-time work, I guess five were spent goggling into my screen and banging my head onto the keyboard.

Continuations are hard to sell. Coroutines and generators, and especially microthreads are easier. All of the above can be implemented without having explicit continuations. But when you have continuations already, you find that the step to these other structures is quite small, and continuations are the way to go. So I'm going to change my marketing strategy and not try any longer to sell the continuations, but their outcome. Continuations will still be there for those who can see the light.

Mertz: There is a joke about American engineers and French engineers. The American team brings a prototype to the French team. The French team's response is: "Well, it works fine in practice; but how will it hold up in theory?" I think the joke is probably meant to poke fun at a "French" style, but in my own mind I completely identify with the "French" reaction. Bracketing any specific national stereotypes in the joke, it is my identification in it that draws me to Stackless. CPython works in practice, but Stackless works in theory! (In other words, the abstract purity of continuations is more interesting to me personally than is the context switch speedups of microthreads, for example).

Tismer: My feeling is a bit similar. After realizing that CPython can be implemented without the C stack involved, I was sure that it must be implemented this way; everything else looks insane to me. CPython already pays for the overhead of frame objects, but it throws all their freedom away by tying them to the C stack. I felt I had to liberate Python. :-)

I started the project in May 1999. Sam Rushing was playing with a hardware coroutine implementation, and a discussion on Python-dev began. Such a stack copying hack would never make it into Python, that was clear. But a portable, clean implementation of coroutines would, possibly. Unfortunately, this is impossible. Steve Majewski gave up five years ago, after he realized that he could not solve this problem without completely rewriting Python.

That was the challenge. I had to find out. Either it is possible, and I would implement it; or it is not, and I would prove the impossibility. Not much later, after first thoughts and attempts, Sam told me about call/cc and how powerful it was. At this time, I had no idea in what way they could be more powerful than coroutines, but I believed him and implemented them; after six or seven times, always a complete rewrite, I understood more.

Ultimately I wanted to create threads at blinding speed, but my primary intent was to find out how far I can reach at all.

Mertz: On the practical side, just what performance improvements is Stackless likely to have? How great are these improvements in the current implementation? How much more is possible with tweaking? What specific sorts of applications are most likely to benefit from Stackless?

Tismer: With the current implementation, there is no large advantage for Stackless over the traditional calling scheme. Normal Python starts a recursion to a new interpreter. Stackless unwinds up to a dispatcher and starts an interpreter from there. This is nearly the same. Real improvements are there for implementations of coroutines and threads. They need to be simulated by classes, or to be real threads in Standard Python, while they can be implemented much more directly with Stackless.

Much more improvement of the core doesn't seem possible without dramatic changes to the opcode set. But a re-implementation, with more built-in support for continuations et. al., can improve the speed of these quite a lot.

Specific applications that might benefit greatly are possibly Swarm simulations, or multiuser games with very many actors performing tiny tasks. One example is the EVE game (see Resources below), which is under development, using Stackless Python.

Mertz: What do you think about incorporating Stackless into the CPython trunk? Is Stackless just as good as an available branch, or does something get better if it becomes the core version?

Tismer: There are arguments for and against it. Against: As long as I'm sitting on the Stackless implementation, it is mine, and I do not need to discuss the hows and whys. But at the same time, I'm struggling (and don't manage) to keep up with CVS. Better to have other people doing this.

Other Python users, who aren't necessarily interested in kinky stuff, won't recognize Stackless at all; just the fact that it happens to be faster, and that the maximum recursion level now is an option and not a hardware limit. And there is another promise for every user: There will be pickleable execution states. That means you can save your program while it is running, send it to a friend, and continue running it.

Finally, I'm all for it, provided that all my stuff makes it into the core; at the same time, I do not want to see a half-baked solution, as has been proposed several times.

Mertz: Any thoughts on future directions for Stackless? Anything new and different expected down the pipeline? Stackless still suffers from some recursions. Will they vanish?

Tismer: Pickling support will be partially implemented. This will be working first for microthreads since they provide the cleanest abstraction at the moment. They are living in a "clean room" where the remaining recursion problem doesn't exist. My final goal is to remove all interpreter recursion from Python. Some parts of Stackless still have recursions, especially all the predefined __xxx__ methods of objects. This is very hard to finalize since we need to change quite a few things, add new opcodes, unroll certain internal calling sequences, and so on.

Resources

LinuxMall.com - BeOpen.com Swallows Python Whole (page 1 of 3)

Python's fearless leader and self-proclaimed "Benevolent Dictator for Life" (BDFL), Guido van Rossum, recently wrote an open letter to the community, proclaiming Python's decision to move the project to new auspices in the Open Source realm.

"Python is growing rapidly. In order to take it to the next level, I've moved with my core development group to a new employer, BeOpen.com," van Rossum wrote. "BeOpen.com is a startup company with a focus on Open Source communities, and an interest in facilitating next-generation application development. It is a natural fit for Python."

BeOpen.com develops online communities for Open Source application users, developers and interested parties through its network of portal Web sites. It also offers support and custom development for Open Source applications specific to major corporations and governmental entities.

Van Rossum created Python in the early 1990s at CWI (the National Research Institute for Mathematics and Computer Science in the Netherlands) in Amsterdam. In 1995, he moved to the United States, and lives in Reston, Virginia.

In the past five years, van Rossum has worked as a researcher at the Corporation for National Research Initiatives (CNRI). He also is technical director of the Python Consortium, a CNRI-hosted international consortium that promotes Python use and development.

Van Rossum has headed up Python's development for the past decade. He said he expects to remain in the position for at least another 10 years and although he "is not worried about getting hit by a bus," he also has announced that he soon will be married and the union between Python and BeOpen.com will give him some time for a honeymoon.

At BeOpen.com, van Rossum is the director of a new development team recently dubbed PythonLabs. The team includes three of his colleagues from CNRI: Fred Drake, Jeremy Hylton, and Barry Warsaw. "Another familiar face will join us shortly: Tim Peters. We have our own Web site where you can read more about us, our plans and our activities. We've also posted a FAQ there specifically about PythonLabs, our transition to BeOpen.com, and what it means for the Python community," he said.

BeOpen Interview with Guido van Rossum [Python creator]

Linux Today

As open source projects go, Python, the "very high level language" developed by Guido Van Rossum 10 years ago, is a prime example of the "scratch your own itch" design philosophy.

Van Rossum, a 44 year old developer who spent much of his collegiate and post- collegiate years working with instructional software languages such as ABC and Pascal, freely admits that it was his annoyance with these languages' real-word performance that drove him to create Python.

"The history of Python came out of the frustration I had with ABC when it wasn't being used for teaching but for day-to-day ad hoc programming," says Van Rossum, who in 1999 received an "Excellence in Programming" award from the Dr. Dobb's Journal for his Python work.

Nearly a decade after releasing the first version of Python, Van Rossum is currently looking at how to bridge the gap between languages that are easy for non-technical users to grasp and languages that are capable of performing industrial-strength computational tasks. Recently, he and other Python developers have joined together to form Computer Programming for Everybody, or CP4E, a project that is currently seeking funds from DARPA, the modern day descendant of the organization that helped build the early Internet. According to Van Rossum, CP4E will explore the way non-technical users engage in computing, in a way that will gain new insights into smoothing the human/machine interaction.

Fast, Free Prototyping in Python by Andrea Provaglio

Though Visual Basic has unbeatable GUI and COM capabilities, Python is an open-source tool with surprising utility for corporate development.

[Apr 02, 2000] Linux Gazette; Exploring parsing and virtual machines with Python


Tutorials

Give Python a try. I think you'll like it. The language itself is very simple (see also discussion of the braces problem below). All the power is found in the extension modules. To learn the language basics should only take a week or so. I think the ease of learning argument is also a powerful argument in favor of Python. I've been programming Perl off and on several years now, but I cannot claim to be a Perl guru -- the language is so huge and so baroque, a really PL/1-style language that I like too ;-). IMHO nobody has any change to master Perl in a week ;-).


Continuations

Guido's thoughts about continuations

Third ACM SIGPLAN Workshop on Continuations

2nd ACM SIGPLAN Workshop on Continuations

Dorai Sitaram, Teach Yourself Scheme in Fixnum Days, http://www.cs.rice.edu/~dorai/t-y-scheme/, September 1998

Andrew W. Appel, Compiling with Continuations, Cambridge University Press, 1992


Jython



Etc

FAIR USE NOTICE This site contains copyrighted material the use of which has not always been specifically authorized by the copyright owner. We are making such material available in our efforts to advance understanding of environmental, political, human rights, economic, democracy, scientific, and social justice issues, etc. We believe this constitutes a 'fair use' of any such copyrighted material as provided for in section 107 of the US Copyright Law. In accordance with Title 17 U.S.C. Section 107, the material on this site is distributed without profit exclusivly for research and educational purposes.   If you wish to use copyrighted material from this site for purposes of your own that go beyond 'fair use', you must obtain permission from the copyright owner. 

ABUSE: IPs or network segments from which we detect a stream of probes might be blocked for no less then 90 days. Multiple types of probes increase this period.  

Society

Groupthink : Two Party System as Polyarchy : Corruption of Regulators : Bureaucracies : Understanding Micromanagers and Control Freaks : Toxic Managers :   Harvard Mafia : Diplomatic Communication : Surviving a Bad Performance Review : Insufficient Retirement Funds as Immanent Problem of Neoliberal Regime : PseudoScience : Who Rules America : Neoliberalism  : The Iron Law of Oligarchy : Libertarian Philosophy

Quotes

War and Peace : Skeptical Finance : John Kenneth Galbraith :Talleyrand : Oscar Wilde : Otto Von Bismarck : Keynes : George Carlin : Skeptics : Propaganda  : SE quotes : Language Design and Programming Quotes : Random IT-related quotesSomerset Maugham : Marcus Aurelius : Kurt Vonnegut : Eric Hoffer : Winston Churchill : Napoleon Bonaparte : Ambrose BierceBernard Shaw : Mark Twain Quotes

Bulletin:

Vol 25, No.12 (December, 2013) Rational Fools vs. Efficient Crooks The efficient markets hypothesis : Political Skeptic Bulletin, 2013 : Unemployment Bulletin, 2010 :  Vol 23, No.10 (October, 2011) An observation about corporate security departments : Slightly Skeptical Euromaydan Chronicles, June 2014 : Greenspan legacy bulletin, 2008 : Vol 25, No.10 (October, 2013) Cryptolocker Trojan (Win32/Crilock.A) : Vol 25, No.08 (August, 2013) Cloud providers as intelligence collection hubs : Financial Humor Bulletin, 2010 : Inequality Bulletin, 2009 : Financial Humor Bulletin, 2008 : Copyleft Problems Bulletin, 2004 : Financial Humor Bulletin, 2011 : Energy Bulletin, 2010 : Malware Protection Bulletin, 2010 : Vol 26, No.1 (January, 2013) Object-Oriented Cult : Political Skeptic Bulletin, 2011 : Vol 23, No.11 (November, 2011) Softpanorama classification of sysadmin horror stories : Vol 25, No.05 (May, 2013) Corporate bullshit as a communication method  : Vol 25, No.06 (June, 2013) A Note on the Relationship of Brooks Law and Conway Law

History:

Fifty glorious years (1950-2000): the triumph of the US computer engineering : Donald Knuth : TAoCP and its Influence of Computer Science : Richard Stallman : Linus Torvalds  : Larry Wall  : John K. Ousterhout : CTSS : Multix OS Unix History : Unix shell history : VI editor : History of pipes concept : Solaris : MS DOSProgramming Languages History : PL/1 : Simula 67 : C : History of GCC developmentScripting Languages : Perl history   : OS History : Mail : DNS : SSH : CPU Instruction Sets : SPARC systems 1987-2006 : Norton Commander : Norton Utilities : Norton Ghost : Frontpage history : Malware Defense History : GNU Screen : OSS early history

Classic books:

The Peter Principle : Parkinson Law : 1984 : The Mythical Man-MonthHow to Solve It by George Polya : The Art of Computer Programming : The Elements of Programming Style : The Unix Hater’s Handbook : The Jargon file : The True Believer : Programming Pearls : The Good Soldier Svejk : The Power Elite

Most popular humor pages:

Manifest of the Softpanorama IT Slacker Society : Ten Commandments of the IT Slackers Society : Computer Humor Collection : BSD Logo Story : The Cuckoo's Egg : IT Slang : C++ Humor : ARE YOU A BBS ADDICT? : The Perl Purity Test : Object oriented programmers of all nations : Financial Humor : Financial Humor Bulletin, 2008 : Financial Humor Bulletin, 2010 : The Most Comprehensive Collection of Editor-related Humor : Programming Language Humor : Goldman Sachs related humor : Greenspan humor : C Humor : Scripting Humor : Real Programmers Humor : Web Humor : GPL-related Humor : OFM Humor : Politically Incorrect Humor : IDS Humor : "Linux Sucks" Humor : Russian Musical Humor : Best Russian Programmer Humor : Microsoft plans to buy Catholic Church : Richard Stallman Related Humor : Admin Humor : Perl-related Humor : Linus Torvalds Related humor : PseudoScience Related Humor : Networking Humor : Shell Humor : Financial Humor Bulletin, 2011 : Financial Humor Bulletin, 2012 : Financial Humor Bulletin, 2013 : Java Humor : Software Engineering Humor : Sun Solaris Related Humor : Education Humor : IBM Humor : Assembler-related Humor : VIM Humor : Computer Viruses Humor : Bright tomorrow is rescheduled to a day after tomorrow : Classic Computer Humor

The Last but not Least


Copyright © 1996-2016 by Dr. Nikolai Bezroukov. www.softpanorama.org was created as a service to the UN Sustainable Development Networking Programme (SDNP) in the author free time. This document is an industrial compilation designed and created exclusively for educational use and is distributed under the Softpanorama Content License.

The site uses AdSense so you need to be aware of Google privacy policy. You you do not want to be tracked by Google please disable Javascript for this site. This site is perfectly usable without Javascript.

Original materials copyright belong to respective owners. Quotes are made for educational purposes only in compliance with the fair use doctrine.

FAIR USE NOTICE This site contains copyrighted material the use of which has not always been specifically authorized by the copyright owner. We are making such material available to advance understanding of computer science, IT technology, economic, scientific, and social issues. We believe this constitutes a 'fair use' of any such copyrighted material as provided by section 107 of the US Copyright Law according to which such material can be distributed without profit exclusively for research and educational purposes.

This is a Spartan WHYFF (We Help You For Free) site written by people for whom English is not a native language. Grammar and spelling errors should be expected. The site contain some broken links as it develops like a living tree...

You can use PayPal to make a contribution, supporting development of this site and speed up access. In case softpanorama.org is down you can use the at softpanorama.info

Disclaimer:

The statements, views and opinions presented on this web page are those of the author (or referenced source) and are not endorsed by, nor do they necessarily reflect, the opinions of the author present and former employers, SDNP or any other organization the author may be associated with. We do not warrant the correctness of the information provided or its fitness for any purpose.

Last modified: November, 16, 2017