Softpanorama

Home Switchboard Unix Administration Red Hat TCP/IP Networks Neoliberalism Toxic Managers
May the source be with you, but remember the KISS principle ;-)
Skepticism and critical thinking is not panacea, but can help to understand the world better

Scriptorama: A Slightly Skeptical View
on Scripting Languages

News Introduction Recommended Books Recommended Links Programming Languages Design Recommended Papers Scripting Languages for Java vs. Pure Java
Software Engineering Anti-OO John Ousterhout Larry Wall Shell Giants Software Prototyping Software Life Cycle Models
Shells AWK Perl Perl Warts and Quirks Python PHP Javascript
Ruby Tcl/Tk R programming language Rexx Lua S-lang JVM-based scripting languages
Pipes Regex Program understanding Beautifiers and Pretty Printers Neatbash -- a simple bash prettyprinter Neatperl -- a simple Perl prettyprinter  
Brooks law Conway Law KISS Principle Featuritis Software Prototyping Unix Component Model  
Programming as a Profession Programming style Language design and programming quotes History Humor Random Findings Etc

This is the central page of the Softpanorama WEB site because I am strongly convinced that the development of scripting languages, not the replication of the efforts of BSD group undertaken by Stallman and Torvalds is the central part of open source. See Scripting languages as VHLL for more details.

 
Ordinarily technology changes fast. But programming languages are different: programming languages are not just technology, but what programmers think in.

They're half technology and half religion. And so the median language, meaning whatever language the median programmer uses, moves as slow as an iceberg.

Paul Graham: Beating the Averages

Libraries are more important that the language.

Donald Knuth


Introduction

A fruitful way to think about language development is to consider it a to be special type of theory building. Peter Naur suggested that programming in general is theory building activity in his 1985 paper "Programming as Theory Building". But idea is especially applicable to compilers and interpreters. What Peter Naur failed to understand was that design of programming languages has religious overtones and sometimes represent an activity, which is pretty close to the process of creating a new, obscure cult ;-). Clueless academics publishing junk papers at obscure conferences are high priests of the church of programming languages. Some, like Niklaus Wirth and Edsger W. Dijkstra, (temporary) reached the status close to (false) prophets :-).

On a deep conceptual level building of a new language is a human way of solving complex problems. That means that complier construction in probably the most underappreciated paradigm of programming of large systems. Much more so then greatly oversold object-oriented programming. OO benefits are greatly overstated.

For users, programming languages distinctly have religious aspects, so decisions about what language to use are often far from being rational and are mainly cultural.  Indoctrination at the university plays a very important role. Recently they were instrumental in making Java a new Cobol.

The second important observation about programming languages is that language per se is just a tiny part of what can be called language programming environment. The latter includes libraries, IDE, books, level of adoption at universities,  popular, important applications written in the language, level of support and key players that support the language on major platforms such as Windows and Linux and other similar things.

A mediocre language with good programming environment can give a run for the money to similar superior in design languages that are just naked.  This is  a story behind success of  Java and PHP. Critical application is also very important and this is a story of success of PHP which is nothing but a bastardatized derivative of Perl (with all the most interesting Perl features surgically removed ;-) adapted to creation of dynamic web sites using so called LAMP stack.

Progress in programming languages has been very uneven and contain several setbacks. Currently this progress is mainly limited to development of so called scripting languages.  Traditional high level languages field is stagnant for many decades.  From 2000 to 2017 we observed the huge sucess of Javascript; Python encroached in Perl territory (including genomics/bioinformatics) and R in turn start squeezing Python in several areas. At the same time Ruby despite initial success remained niche language.  PHP still holds its own in web-site design.

Some observations about scripting language design and  usage

At the same time there are some mysterious, unanswered question about factors that help the particular scripting language to increase its user base, or fail in popularity. Among them:

Nothing succeed like success

Those are difficult questions to answer without some way of classifying languages into different categories. Several such classifications exists. First of all like with natural languages, the number of people who speak a given language is a tremendous force that can overcome any real of perceived deficiencies of the language. In programming languages, like in natural languages nothing succeed like success.

The second interesting category is number of applications written in particular language that became part of Linux or, at least, are including in standard RHEL/FEDORA/CENTOS or Debian/Ubuntu repository.

The third relevant category is the number and quality of books for the particular language.

Complexity Curse

History of programming languages raises interesting general questions about the limit of complexity of programming languages. There is strong historical evidence that a language with simpler core, or even simplistic core Basic, Pascal) have better chances to acquire high level of popularity.

The underlying fact here probably is that most programmers are at best mediocre and such programmers tend on intuitive level to avoid more complex, more rich languages and prefer, say, Pascal to PL/1 and PHP to Perl. Or at least avoid it on a particular phase of language development (C++ is not simpler language then PL/1, but was widely adopted because of the progress of hardware, availability of compilers and not the least, because it was associated with OO exactly at the time OO became a mainstream fashion).

Complex non-orthogonal languages can succeed only as a result of a long period of language development (which usually adds complexly -- just compare Fortran IV with Fortran 99; or PHP 3 with PHP 5 ) from a smaller core. Attempts to ride some fashionable new trend extending existing popular language to this new "paradigm" also proved to be relatively successful (OO programming in case of C++, which is a superset of C).

Historically, few complex languages were successful (PL/1, Ada, Perl, C++), but even if they were successful, their success typically was temporary rather then permanent  (PL/1, Ada, Perl). As Professor Wilkes noted   (iee90):

Things move slowly in the computer language field but, over a sufficiently long period of time, it is possible to discern trends. In the 1970s, there was a vogue among system programmers for BCPL, a typeless language. This has now run its course, and system programmers appreciate some typing support. At the same time, they like a language with low level features that enable them to do things their way, rather than the compiler’s way, when they want to.

They continue, to have a strong preference for a lean language. At present they tend to favor C in its various versions. For applications in which flexibility is important, Lisp may be said to have gained strength as a popular programming language.

Further progress is necessary in the direction of achieving modularity. No language has so far emerged which exploits objects in a fully satisfactory manner, although C++ goes a long way. ADA was progressive in this respect, but unfortunately it is in the process of collapsing under its own great weight.

ADA is an example of what can happen when an official attempt is made to orchestrate technical advances. After the experience with PL/1 and ALGOL 68, it should have been clear that the future did not lie with massively large languages.

I would direct the reader’s attention to Modula-3, a modest attempt to build on the appeal and success of Pascal and Modula-2 [12].

Complexity of the compiler/interpreter also matter as it affects portability: this is one thing that probably doomed PL/1 (and later Ada), although those days a new language typically come with open source compiler (or in case of scripting languages, an interpreter) and this is less of a problem.

Here is an interesting take on language design from the preface to The D programming language book 9D language failed to achieve any significant level of popularity):

Programming language design seeks power in simplicity and, when successful, begets beauty.

Choosing the trade-offs among contradictory requirements is a difficult task that requires good taste from the language designer as much as mastery of theoretical principles and of practical implementation matters. Programming language design is software-engineering-complete.

D is a language that attempts to consistently do the right thing within the constraints it chose: system-level access to computing resources, high performance, and syntactic similarity with C-derived languages. In trying to do the right thing, D sometimes stays with tradition and does what other languages do, and other times it breaks tradition with a fresh, innovative solution. On occasion that meant revisiting the very constraints that D ostensibly embraced. For example, large program fragments or indeed entire programs can be written in a well-defined memory-safe subset of D, which entails giving away a small amount of system-level access for a large gain in program debuggability.

You may be interested in D if the following values are important to you:

The role of fashion

At the initial, the most difficult stage of language development the language should solve an important problem that was inadequately solved by currently popular languages.  But at the same time the language has few chances to succeed unless it perfectly fits into the current software fashion. This "fashion factor" is probably as important as several other factors combined. With the notable exclusion of "language sponsor" factor.  The latter can make or break the language.

Like in woman dress fashion rules in language design.  And with time this trend became more and more pronounced.  A new language should simultaneously represent the current fashionable trend.  For example OO-programming was a visit card into the world of "big, successful languages" since probably early 90th (C++, Java, Python).  Before that "structured programming" and "verification" (Pascal, Modula) played similar role.

Programming environment and the role of "powerful sponsor" in language success

PL/1, Java, C#, Ada, Python are languages that had powerful sponsors. Pascal, Basic, Forth, partially Perl (O'Reilly was a sponsor for a short period of time)  are examples of the languages that had no such sponsor during the initial period of development.  C and C++ are somewhere in between.

But language itself is not enough. Any language now need a "programming environment" which consists of a set of libraries, debugger and other tools (make tool, lint, pretty-printer, etc). The set of standard" libraries and debugger are probably two most important elements. They cost  lot of time (or money) to develop and here the role of powerful sponsor is difficult to underestimate.

While this is not the necessary condition for becoming popular, it really helps: other things equal the weight of the sponsor of the language does matter. For example Java, being a weak, inconsistent language (C-- with garbage collection and OO) was pushed through the throat on the strength of marketing and huge amount of money spend on creating Java programming environment. 

The same was partially true for  C# and Python. That's why Python, despite its "non-Unix" origin is more viable scripting language now then, say, Perl (which is better integrated with Unix and has pretty innovative for scripting languages support of pointers and regular expressions), or Ruby (which has support of coroutines from day 1, not as "bolted on" feature like in Python).

Like in political campaigns, negative advertizing also matter. For example Perl suffered greatly from blackmail comparing programs in it with "white noise". And then from withdrawal of O'Reilly from the role of sponsor of the language (although it continue to milk that Perl book publishing franchise ;-)

People proved to be pretty gullible and in this sense language marketing is not that different from woman clothing marketing :-)

Language level and success

One very important classification of programming languages is based on so called the level of the language.  Essentially after there is at least one language that is successful on a given level, the success of other languages on the same level became more problematic. Higher chances for success are for languages that have even slightly higher, but still higher level then successful predecessors.

The level of the language informally can be described as the number of statements (or, more correctly, the number of  lexical units (tokens)) needed to write a solution of a particular problem in one language versus another. This way we can distinguish several levels of programming languages:

 "Nanny languages" vs "Sharp razor" languages

Some people distinguish between "nanny languages" and "sharp razor" languages. The latter do not attempt to protect user from his errors while the former usually go too far... Right compromise is extremely difficult to find.

For example, I consider the explicit availability of pointers as an important feature of the language that greatly increases its expressive power and far outweighs risks of errors in hands of unskilled practitioners.  In other words attempts to make the language "safer" often misfire.

Expressive style of the languages

Another useful typology is based in expressive style of the language:

Those categories are not pure and somewhat overlap. For example, it's possible to program in an object-oriented style in C, or even assembler. Some scripting languages like Perl have built-in regular expressions engines that are a part of the language so they have functional component despite being procedural. Some relatively low level languages (Algol-style languages) implement garbage collection. A good example is Java. There are scripting languages that compile into common language framework which was designed for high level languages. For example, Iron Python compiles into .Net.

Weak correlation between quality of design and popularity

Popularity of the programming languages is not strongly connected to their quality. Some languages that look like a collection of language designer blunders (PHP, Java ) became quite popular. Java became  a new Cobol and PHP dominates dynamic Web sites construction. The dominant technology for such Web sites is often called LAMP, which means Linux - Apache -MySQL- PHP. Being a highly simplified but badly constructed subset of Perl ( kind of new Basic for dynamic Web sites) PHP provides the most depressing experience. I was unpleasantly surprised when I had learnt that the Wikipedia engine was rewritten in PHP from Perl some time ago, but this fact quite illustrates the trend.  The number of mediocre programmer outweigh  the number of talented programmers by factor of 100 or higher.

So language design quality has little to do with the language success in the marketplace. Simpler languages have more wide appeal as success of PHP (which at the beginning was at the expense of Perl) suggests. In addition much depends whether the language has powerful sponsor like was the case with Java (Sun and IBM), PHO (Facebook). This is partially true for Python (Google) but it was after the designers of the language spend many years fighting for survival. 

Progress in programming languages has been very uneven and contain several setbacks like Java and PHP (and partially C++). Currently this progress is usually associated with scripting languages. History of programming languages raises interesting general questions about "laws" of programming language design. First let's reproduce several notable quotes:

  1. Knuth law of optimization: "Premature optimization is the root of all evil (or at least most of it) in programming." - Donald Knuth
  2. "Greenspun's Tenth Rule of Programming: any sufficiently complicated C or Fortran program contains an ad hoc informally-specified bug-ridden slow implementation of half of Common Lisp." - Phil Greenspun
  3. "The key to performance is elegance, not battalions of special cases." - Jon Bentley and Doug McIlroy
  4. "Some may say Ruby is a bad rip-off of Lisp or Smalltalk, and I admit that. But it is nicer to ordinary people." - Matz, LL2
  5. Most papers in computer science describe how their author learned what someone else already knew. - Peter Landin
  6. "The only way to learn a new programming language is by writing programs in it." - Kernighan and Ritchie
  7. "If I had a nickel for every time I've written "for (i = 0; i < N; i++)" in C, I'd be a millionaire." - Mike Vanier
  8. "Language designers are not intellectuals. They're not as interested in thinking as you might hope. They just want to get a language done and start using it." - Dave Moon
  9. "Don't worry about what anybody else is going to do. The best way to predict the future is to invent it." - Alan Kay
  10. "Programs must be written for people to read, and only incidentally for machines to execute." - Abelson & Sussman, SICP, preface to the first edition

Please note that one thing is to read language manual and appreciate how good the concepts are, and another to bet your project on a new, unproved language without good debuggers, manuals and, what is very important, libraries. Debugger is very important but standard libraries are crucial: they represent a factor that makes or breaks new languages.

In this sense languages are much like cars. For many people car is the thing that they use get to work and shopping mall and they are not very interesting is engine inline or V-type and the use of fuzzy logic in the transmission. What they care is safety, reliability, mileage, insurance and the size of trunk. In this sense "Worse is better" is very true. I already mentioned the importance of the debugger. The other important criteria is quality and availability of libraries. Actually libraries are what make 80% of the usability of the language, moreover in a sense libraries are more important than the language...

A popular belief that scripting is "unsafe" or "second rate" or "prototype" solution is completely wrong. If a project had died than it does not matter what was the implementation language, so for any successful project and tough schedules scripting language (especially in dual scripting language+C combination, for example TCL+C) is an optimal blend that for a large class of tasks. Such an approach helps to separate architectural decisions from implementation details much better that any OO model does.

Moreover even for tasks that handle a fair amount of computations and data (computationally intensive tasks) such languages as Python and Perl are often (but not always !) competitive with C++, C# and, especially, Java.

The second important observation about programming languages is that language per se is just a tiny part of what can be called language programming environment. the latter includes libraries, IDE, books, level of adoption at universities, popular, important applications written in the language, level of support and key players that support the language on major platforms such as Windows and Linux and other similar things. A mediocre language with good programming environment can give a run for the money to similar superior in design languages that are just naked. This is a story behind success of Java. Critical application is also very important and this is a story of success of PHP which is nothing but a bastardatized derivative of Perl (with all most interesting Perl features removed ;-) adapted to creation of dynamic web sites using so called LAMP stack.

History of programming languages raises interesting general questions about the limit of complexity of programming languages. There is strong historical evidence that languages with simpler core, or even simplistic core has more chanced to acquire high level of popularity. The underlying fact here probably is that most programmers are at best mediocre and such programmer tend on intuitive level to avoid more complex, more rich languages like, say, PL/1 and Perl. Or at least avoid it on a particular phase of language development (C++ is not simpler language then PL/1, but was widely adopted because OO became a fashion). Complex non-orthogonal languages can succeed only as a result on long period of language development from a smaller core or with the banner of some fashionable new trend (OO programming in case of C++).

Programming Language Development Timeline

Here is modified from Byte the timeline of Programming Languages (for the original see BYTE.com September 1995 / 20th Anniversary /)

Forties

ca. 1946


1949

Fifties


1951


1952

1957


1958


1959

Sixties


1960


1962


1963


1964


1965


1966


1967



1969

Seventies


1970


1972


1974


1975


1976


1977


1978


1979

Eighties


1980


1981


1982


1983


1984


1985


1986


1987


1988


1989

Nineties


1990


1991


1992


1993


1994


1995


1996


1997


2000


2006

2007 

2008:

2009:

2011

2017:

Special note on Scripting languages

Scripting helps to avoid OO trap that is pushed by
  "a hoard of practically illiterate researchers
publishing crap papers in junk conferences."

Despite the fact that scripting languages are really important computer science phenomena, they are usually happily ignored in university curriculums.  Students are usually indoctrinated (or in less politically correct terms  "brainwashed")  in Java and OO programming ;-)

This site tries to give scripting languages proper emphasis and  promotes scripting languages as an alternative to mainstream reliance on "Java as a new Cobol" approach for software development. Please read my introduction to the topic that was recently converted into the article: A Slightly Skeptical View on Scripting Languages.  

The tragedy of scripting language designer is that there is no way to overestimate the level of abuse of any feature of the language.  Half of the programmers by definition is below average and it is this half that matters most in enterprise environment.  In a way the higher is the level of programmer, the less relevant for him are limitations of the language. That's why statements like "Perl is badly suitable for large project development" are plain vanilla silly. With proper discipline it is perfectly suitable and programmers can be more productive with Perl than with Java. The real question is "What is the team quality and quantity?".  

Scripting is a part of Unix cultural tradition and Unix was the initial development platform for most of mainstream scripting languages with the exception of REXX. But they are portable and now all can be used in Windows and other OSes.

List of Softpanorama pages related to scripting languages

Standard topics

Main Representatives of the family Related topics History Etc

Different scripting languages provide different level of integration with base OS API (for example, Unix or Windows). For example Iron Python compiles into .Net and provides pretty high level of integration with Windows. The same is true about Perl and Unix: almost all Unix system calls are available directly from Perl. Moreover Perl integrates most of Unix API in a very natural way, making it perfect replacement of shell for coding complex scripts. It also have very good debugger. The latter is weak point of shells like bash and ksh93

Unix proved that treating everything like a file is a powerful OS paradigm. In a similar way scripting languages proved that "everything is a string" is also an extremely powerful programming paradigm.

Unix proved that treating everything like a file is a powerful OS paradigm. In a similar way scripting languages proved that "everything is a string" is also extremely powerful programming paradigm.

There are also several separate pages devoted to scripting in different applications. The main emphasis is on shells and Perl. Right now I am trying to convert my old Perl lecture notes into a eBook Nikolai Bezroukov. Introduction to Perl for Unix System Administrators.

Along with pages devoted to major scripting languages this site has many pages devoted to scripting in different applications.  There are more then a dozen of "Perl/Scripting tools for a particular area" type of pages. The most well developed and up-to-date pages of this set are probably Shells and Perl. This page main purpose is to follow the changes in programming practices that can be called the "rise of  scripting," as predicted in the famous John Ousterhout article Scripting: Higher Level Programming for the 21st Century in IEEE COMPUTER (1998). In this brilliant paper he wrote:

...Scripting languages such as Perl and Tcl represent a very different style of programming than system programming languages such as C or Java. Scripting languages are designed for "gluing" applications; they use typeless approaches to achieve a higher level of programming and more rapid application development than system programming languages. Increases in computer speed and changes in the application mix are making scripting languages more and more important for applications of the future.

...Scripting languages and system programming languages are complementary, and most major computing platforms since the 1960's have provided both kinds of languages. The languages are typically used together in component frameworks, where components are created with system programming languages and glued together with scripting languages. However, several recent trends, such as faster machines, better scripting languages, the increasing importance of graphical user interfaces and component architectures, and the growth of the Internet, have greatly increased the applicability of scripting languages. These trends will continue over the next decade, with more and more new applications written entirely in scripting languages and system programming languages used primarily for creating components.

My e-book Portraits of Open Source Pioneers contains several chapters on scripting (most are in early draft stage) that expand on this topic. 

The reader must understand that the treatment of the scripting languages in press, and especially academic press is far from being fair: entrenched academic interests often promote old or commercially supported paradigms until they retire, so change of paradigm often is possible only with the change of generations. And people tend to live longer those days... Please also be aware that even respectable academic magazines like Communications of ACM and IEEE Software often promote "Cargo cult software engineering" like Capability Maturity (CMM) model.

Dr. Nikolai Bezroukov


Top Visited
Switchboard
Latest
Past week
Past month

NEWS CONTENTS

Old News ;-)

2007 2006 2005 2004 2003 2002 2001 2000 1999

[Dec 01, 2019] How can I export all subs in a Perl package?

Jan 01, 2009 | stackoverflow.com

Ask Question Asked 10 years, 7 months ago Active 3 years, 5 months ago Viewed 18k times


Ville M ,

I would like to expose all subs into my namespace without having to list them one at a time:
@EXPORT = qw( firstsub secondsub third sub etc );

Using fully qualified names would require bunch of change to existing code so I'd rather not do that.

Is there @EXPORT_ALL?

I think documentation says it's a bad idea, but I'd like to do it anyway, or at least know how.

To answer Jon's why: right now for quick refactoring I want to move of bunch of subs into their own package with least hassle and code changes to the existing scripts (where those subs are currenty used and often repeated).

Also, mostly, I was just curious. (since it seemed like that Exporter might as well have that as standard feature, but somewhat surprisingly based on answers so far it doesn't)

brian d foy , 2009-04-08 23:58:35

Don't do any exporting at all, and don't declare a package name in your library. Just load the file with require and everything will be in the current package. Easy peasy.

Michael Carman , 2009-04-09 00:15:10

Don't. But if you really want to... write a custom import that walks the symbol table and export all the named subroutines.
# Export all subs in package. Not for use in production code!
sub import {
    no strict 'refs';

    my $caller = caller;

    while (my ($name, $symbol) = each %{__PACKAGE__ . '::'}) {
        next if      $name eq 'BEGIN';   # don't export BEGIN blocks
        next if      $name eq 'import';  # don't export this sub
        next unless *{$symbol}{CODE};    # export subs only

        my $imported = $caller . '::' . $name;
        *{ $imported } = \*{ $symbol };
    }
}

Chas. Owens ,

Warning, the code following is as bad an idea as exporting everything:
package Expo;

use base "Exporter";

seek DATA, 0, 0; #move DATA back to package

#read this file looking for sub names
our @EXPORT = map { /^sub\s+([^({\s]+)/ ? $1 : () } <DATA>;

my $sub = sub {}; #make sure anon funcs aren't grabbed

sub foo($) {
    print shift, "\n";
}

sub bar ($) {
    print shift, "\n";
}

sub baz{
    print shift,"\n";
}

sub quux {
    print shift,"\n";
}

1;

__DATA__

Here is the some code that uses the module:

#!/usr/bin/perl

use strict;
use warnings;

use Expo;

print map { "[$_]\n" } @Expo::EXPORT;

foo("foo");
bar("bar");
baz("baz");
quux("quux");

And here is its output:

[foo]
[bar]
[baz]
[quux]
foo
bar
baz
quux

Jon Ericson , 2009-04-08 22:33:36

You can always call subroutines in there fully-specified form:
MyModule::firstsub();

For modules I write internally, I find this convention works fairly well. It's a bit more typing, but tends to be better documentation.

Take a look at perldoc perlmod for more information about what you are trying to accomplish.

More generally, you could look at Exporter 's code and see how it uses glob aliasing. Or you can examine your module's namespace and export each subroutine. (I don't care to search for how to do that at the moment, but Perl makes this fairly easy.) Or you could just stick your subroutines in the main package:

 package main;
 sub firstsub() { ... }

(I don't think that's a good idea, but you know better than I do what you are trying to accomplish.)

There's nothing wrong with doing this provided you know what you are doing and aren't just trying to avoid thinking about your interface to the outside world.

ysth , 2009-04-09 01:29:04

Perhaps you would be interested in one of the Export* modules on CPAN that lets you mark subs as exportable simply by adding an attribute to the sub definition? (Don't remember which one it was, though.)

echo , 2014-10-11 18:23:01

https://metacpan.org/pod/Exporter::Auto

Exporter::Auto. this is all you need.

Tero Niemi , 2013-04-02 00:32:25

Although it is not usually wise to dump all sub s from module into the caller namespace, it is sometimes useful (and more DRY!) to automatically generate @EXPORT_OK and %EXPORT_TAGS variables.

The easiest method is to extend the Exporter. A simple example is something like this:

package Exporter::AutoOkay;
#
#   Automatically add all subroutines from caller package into the
#   @EXPORT_OK array. In the package use like Exporter, f.ex.:
#
#       use parent 'Exporter::AutoOkay';
#
use warnings;
use strict;
no strict 'refs';

require Exporter;

sub import {
    my $package = $_[0].'::';

    # Get the list of exportable items
    my @export_ok = (@{$package.'EXPORT_OK'});

    # Automatically add all subroutines from package into the list
    foreach (keys %{$package}) {
        next unless defined &{$package.$_};
        push @export_ok, $_;
    }

    # Set variable ready for Exporter
    @{$package.'EXPORT_OK'} = @export_ok;

    # Let Exporter do the rest
    goto &Exporter::import;
}

1;

Note the use of goto that removes us from the caller stack.

A more complete example can be found here: http://pastebin.com/Z1QWzcpZ It automatically generates tag groups from subroutine prefixes.

Sérgio , 2013-11-14 21:38:06

case 1

Library is :

package mycommon;

use strict;
use warnings;

sub onefunctionthatyoumadeonlibary() {
}
1;

you can use it, calling common:: :

#!/usr/bin/perl
use strict;
use warnings;
use mycommon;

common::onefunctionthatyoumadeonlibary()
case 2

Library is , yousimple export them :

package mycommon;

use strict;
use warnings;

use base 'Exporter';

our @EXPORT = qw(onefunctionthatyoumadeonlibary);
sub onefunctionthatyoumadeonlibary() {
}
1;

use it in same "namespace":

#!/usr/bin/perl
use strict;
use warnings;
use mycommon qw(onefunctionthatyoumadeonlibary);

onefunctionthatyoumadeonlibary()

Also we can do a mix of this two cases , we can export more common functions to use it without calling the packages name and other functions that we only call it with package name and that ones don't need to be exported.

> ,

You will have to do some typeglob munging. I describe something similar here:

Is there a way to "use" a single file that in turn uses multiple others in Perl?

The import routine there should do exactly what you want -- just don't import any symbols into your own namespace.

Ville M ,

I would like to expose all subs into my namespace without having to list them one at a time:
@EXPORT = qw( firstsub secondsub third sub etc );

Using fully qualified names would require bunch of change to existing code so I'd rather not do that.

Is there @EXPORT_ALL?

I think documentation says it's a bad idea, but I'd like to do it anyway, or at least know how.

To answer Jon's why: right now for quick refactoring I want to move of bunch of subs into their own package with least hassle and code changes to the existing scripts (where those subs are currenty used and often repeated).

Also, mostly, I was just curious. (since it seemed like that Exporter might as well have that as standard feature, but somewhat surprisingly based on answers so far it doesn't)

brian d foy , 2009-04-08 23:58:35

Don't do any exporting at all, and don't declare a package name in your library. Just load the file with require and everything will be in the current package. Easy peasy.

Michael Carman , 2009-04-09 00:15:10

Don't. But if you really want to... write a custom import that walks the symbol table and export all the named subroutines.
# Export all subs in package. Not for use in production code!
sub import {
    no strict 'refs';

    my $caller = caller;

    while (my ($name, $symbol) = each %{__PACKAGE__ . '::'}) {
        next if      $name eq 'BEGIN';   # don't export BEGIN blocks
        next if      $name eq 'import';  # don't export this sub
        next unless *{$symbol}{CODE};    # export subs only

        my $imported = $caller . '::' . $name;
        *{ $imported } = \*{ $symbol };
    }
}

Chas. Owens ,

Warning, the code following is as bad an idea as exporting everything:
package Expo;

use base "Exporter";

seek DATA, 0, 0; #move DATA back to package

#read this file looking for sub names
our @EXPORT = map { /^sub\s+([^({\s]+)/ ? $1 : () } <DATA>;

my $sub = sub {}; #make sure anon funcs aren't grabbed

sub foo($) {
    print shift, "\n";
}

sub bar ($) {
    print shift, "\n";
}

sub baz{
    print shift,"\n";
}

sub quux {
    print shift,"\n";
}

1;

__DATA__

Here is the some code that uses the module:

#!/usr/bin/perl

use strict;
use warnings;

use Expo;

print map { "[$_]\n" } @Expo::EXPORT;

foo("foo");
bar("bar");
baz("baz");
quux("quux");

And here is its output:

[foo]
[bar]
[baz]
[quux]
foo
bar
baz
quux

Jon Ericson , 2009-04-08 22:33:36

You can always call subroutines in there fully-specified form:
MyModule::firstsub();

For modules I write internally, I find this convention works fairly well. It's a bit more typing, but tends to be better documentation.

Take a look at perldoc perlmod for more information about what you are trying to accomplish.

More generally, you could look at Exporter 's code and see how it uses glob aliasing. Or you can examine your module's namespace and export each subroutine. (I don't care to search for how to do that at the moment, but Perl makes this fairly easy.) Or you could just stick your subroutines in the main package:

 package main;
 sub firstsub() { ... }

(I don't think that's a good idea, but you know better than I do what you are trying to accomplish.)

There's nothing wrong with doing this provided you know what you are doing and aren't just trying to avoid thinking about your interface to the outside world.

ysth , 2009-04-09 01:29:04

Perhaps you would be interested in one of the Export* modules on CPAN that lets you mark subs as exportable simply by adding an attribute to the sub definition? (Don't remember which one it was, though.)

echo , 2014-10-11 18:23:01

https://metacpan.org/pod/Exporter::Auto

Exporter::Auto. this is all you need.

Tero Niemi , 2013-04-02 00:32:25

Although it is not usually wise to dump all sub s from module into the caller namespace, it is sometimes useful (and more DRY!) to automatically generate @EXPORT_OK and %EXPORT_TAGS variables.

The easiest method is to extend the Exporter. A simple example is something like this:

package Exporter::AutoOkay;
#
#   Automatically add all subroutines from caller package into the
#   @EXPORT_OK array. In the package use like Exporter, f.ex.:
#
#       use parent 'Exporter::AutoOkay';
#
use warnings;
use strict;
no strict 'refs';

require Exporter;

sub import {
    my $package = $_[0].'::';

    # Get the list of exportable items
    my @export_ok = (@{$package.'EXPORT_OK'});

    # Automatically add all subroutines from package into the list
    foreach (keys %{$package}) {
        next unless defined &{$package.$_};
        push @export_ok, $_;
    }

    # Set variable ready for Exporter
    @{$package.'EXPORT_OK'} = @export_ok;

    # Let Exporter do the rest
    goto &Exporter::import;
}

1;

Note the use of goto that removes us from the caller stack.

A more complete example can be found here: http://pastebin.com/Z1QWzcpZ It automatically generates tag groups from subroutine prefixes.

Sérgio , 2013-11-14 21:38:06

case 1

Library is :

package mycommon;

use strict;
use warnings;

sub onefunctionthatyoumadeonlibary() {
}
1;

you can use it, calling common:: :

#!/usr/bin/perl
use strict;
use warnings;
use mycommon;

common::onefunctionthatyoumadeonlibary()
case 2

Library is , yousimple export them :

package mycommon;

use strict;
use warnings;

use base 'Exporter';

our @EXPORT = qw(onefunctionthatyoumadeonlibary);
sub onefunctionthatyoumadeonlibary() {
}
1;

use it in same "namespace":

#!/usr/bin/perl
use strict;
use warnings;
use mycommon qw(onefunctionthatyoumadeonlibary);

onefunctionthatyoumadeonlibary()

Also we can do a mix of this two cases , we can export more common functions to use it without calling the packages name and other functions that we only call it with package name and that ones don't need to be exported.

> ,

You will have to do some typeglob munging. I describe something similar here:

Is there a way to "use" a single file that in turn uses multiple others in Perl?

The import routine there should do exactly what you want -- just don't import any symbols into your own namespace.

Ville M ,

I would like to expose all subs into my namespace without having to list them one at a time:
@EXPORT = qw( firstsub secondsub third sub etc );

Using fully qualified names would require bunch of change to existing code so I'd rather not do that.

Is there @EXPORT_ALL?

I think documentation says it's a bad idea, but I'd like to do it anyway, or at least know how.

To answer Jon's why: right now for quick refactoring I want to move of bunch of subs into their own package with least hassle and code changes to the existing scripts (where those subs are currenty used and often repeated).

Also, mostly, I was just curious. (since it seemed like that Exporter might as well have that as standard feature, but somewhat surprisingly based on answers so far it doesn't)

brian d foy , 2009-04-08 23:58:35

Don't do any exporting at all, and don't declare a package name in your library. Just load the file with require and everything will be in the current package. Easy peasy.

Michael Carman , 2009-04-09 00:15:10

Don't. But if you really want to... write a custom import that walks the symbol table and export all the named subroutines.
# Export all subs in package. Not for use in production code!
sub import {
    no strict 'refs';

    my $caller = caller;

    while (my ($name, $symbol) = each %{__PACKAGE__ . '::'}) {
        next if      $name eq 'BEGIN';   # don't export BEGIN blocks
        next if      $name eq 'import';  # don't export this sub
        next unless *{$symbol}{CODE};    # export subs only

        my $imported = $caller . '::' . $name;
        *{ $imported } = \*{ $symbol };
    }
}

Chas. Owens ,

Warning, the code following is as bad an idea as exporting everything:
package Expo;

use base "Exporter";

seek DATA, 0, 0; #move DATA back to package

#read this file looking for sub names
our @EXPORT = map { /^sub\s+([^({\s]+)/ ? $1 : () } <DATA>;

my $sub = sub {}; #make sure anon funcs aren't grabbed

sub foo($) {
    print shift, "\n";
}

sub bar ($) {
    print shift, "\n";
}

sub baz{
    print shift,"\n";
}

sub quux {
    print shift,"\n";
}

1;

__DATA__

Here is the some code that uses the module:

#!/usr/bin/perl

use strict;
use warnings;

use Expo;

print map { "[$_]\n" } @Expo::EXPORT;

foo("foo");
bar("bar");
baz("baz");
quux("quux");

And here is its output:

[foo]
[bar]
[baz]
[quux]
foo
bar
baz
quux

Jon Ericson , 2009-04-08 22:33:36

You can always call subroutines in there fully-specified form:
MyModule::firstsub();

For modules I write internally, I find this convention works fairly well. It's a bit more typing, but tends to be better documentation.

Take a look at perldoc perlmod for more information about what you are trying to accomplish.

More generally, you could look at Exporter 's code and see how it uses glob aliasing. Or you can examine your module's namespace and export each subroutine. (I don't care to search for how to do that at the moment, but Perl makes this fairly easy.) Or you could just stick your subroutines in the main package:

 package main;
 sub firstsub() { ... }

(I don't think that's a good idea, but you know better than I do what you are trying to accomplish.)

There's nothing wrong with doing this provided you know what you are doing and aren't just trying to avoid thinking about your interface to the outside world.

ysth , 2009-04-09 01:29:04

Perhaps you would be interested in one of the Export* modules on CPAN that lets you mark subs as exportable simply by adding an attribute to the sub definition? (Don't remember which one it was, though.)

echo , 2014-10-11 18:23:01

https://metacpan.org/pod/Exporter::Auto

Exporter::Auto. this is all you need.

Tero Niemi , 2013-04-02 00:32:25

Although it is not usually wise to dump all sub s from module into the caller namespace, it is sometimes useful (and more DRY!) to automatically generate @EXPORT_OK and %EXPORT_TAGS variables.

The easiest method is to extend the Exporter. A simple example is something like this:

package Exporter::AutoOkay;
#
#   Automatically add all subroutines from caller package into the
#   @EXPORT_OK array. In the package use like Exporter, f.ex.:
#
#       use parent 'Exporter::AutoOkay';
#
use warnings;
use strict;
no strict 'refs';

require Exporter;

sub import {
    my $package = $_[0].'::';

    # Get the list of exportable items
    my @export_ok = (@{$package.'EXPORT_OK'});

    # Automatically add all subroutines from package into the list
    foreach (keys %{$package}) {
        next unless defined &{$package.$_};
        push @export_ok, $_;
    }

    # Set variable ready for Exporter
    @{$package.'EXPORT_OK'} = @export_ok;

    # Let Exporter do the rest
    goto &Exporter::import;
}

1;

Note the use of goto that removes us from the caller stack.

A more complete example can be found here: http://pastebin.com/Z1QWzcpZ It automatically generates tag groups from subroutine prefixes.

Sérgio , 2013-11-14 21:38:06

case 1

Library is :

package mycommon;

use strict;
use warnings;

sub onefunctionthatyoumadeonlibary() {
}
1;

you can use it, calling common:: :

#!/usr/bin/perl
use strict;
use warnings;
use mycommon;

common::onefunctionthatyoumadeonlibary()
case 2

Library is , yousimple export them :

package mycommon;

use strict;
use warnings;

use base 'Exporter';

our @EXPORT = qw(onefunctionthatyoumadeonlibary);
sub onefunctionthatyoumadeonlibary() {
}
1;

use it in same "namespace":

#!/usr/bin/perl
use strict;
use warnings;
use mycommon qw(onefunctionthatyoumadeonlibary);

onefunctionthatyoumadeonlibary()

Also we can do a mix of this two cases , we can export more common functions to use it without calling the packages name and other functions that we only call it with package name and that ones don't need to be exported.

> ,

You will have to do some typeglob munging. I describe something similar here:

Is there a way to "use" a single file that in turn uses multiple others in Perl?

The import routine there should do exactly what you want -- just don't import any symbols into your own namespace.

Ville M ,

I would like to expose all subs into my namespace without having to list them one at a time:
@EXPORT = qw( firstsub secondsub third sub etc );

Using fully qualified names would require bunch of change to existing code so I'd rather not do that.

Is there @EXPORT_ALL?

I think documentation says it's a bad idea, but I'd like to do it anyway, or at least know how.

To answer Jon's why: right now for quick refactoring I want to move of bunch of subs into their own package with least hassle and code changes to the existing scripts (where those subs are currenty used and often repeated).

Also, mostly, I was just curious. (since it seemed like that Exporter might as well have that as standard feature, but somewhat surprisingly based on answers so far it doesn't)

brian d foy , 2009-04-08 23:58:35

Don't do any exporting at all, and don't declare a package name in your library. Just load the file with require and everything will be in the current package. Easy peasy.

Michael Carman , 2009-04-09 00:15:10

Don't. But if you really want to... write a custom import that walks the symbol table and export all the named subroutines.
# Export all subs in package. Not for use in production code!
sub import {
    no strict 'refs';

    my $caller = caller;

    while (my ($name, $symbol) = each %{__PACKAGE__ . '::'}) {
        next if      $name eq 'BEGIN';   # don't export BEGIN blocks
        next if      $name eq 'import';  # don't export this sub
        next unless *{$symbol}{CODE};    # export subs only

        my $imported = $caller . '::' . $name;
        *{ $imported } = \*{ $symbol };
    }
}

Chas. Owens ,

Warning, the code following is as bad an idea as exporting everything:
package Expo;

use base "Exporter";

seek DATA, 0, 0; #move DATA back to package

#read this file looking for sub names
our @EXPORT = map { /^sub\s+([^({\s]+)/ ? $1 : () } <DATA>;

my $sub = sub {}; #make sure anon funcs aren't grabbed

sub foo($) {
    print shift, "\n";
}

sub bar ($) {
    print shift, "\n";
}

sub baz{
    print shift,"\n";
}

sub quux {
    print shift,"\n";
}

1;

__DATA__

Here is the some code that uses the module:

#!/usr/bin/perl

use strict;
use warnings;

use Expo;

print map { "[$_]\n" } @Expo::EXPORT;

foo("foo");
bar("bar");
baz("baz");
quux("quux");

And here is its output:

[foo]
[bar]
[baz]
[quux]
foo
bar
baz
quux

Jon Ericson , 2009-04-08 22:33:36

You can always call subroutines in there fully-specified form:
MyModule::firstsub();

For modules I write internally, I find this convention works fairly well. It's a bit more typing, but tends to be better documentation.

Take a look at perldoc perlmod for more information about what you are trying to accomplish.

More generally, you could look at Exporter 's code and see how it uses glob aliasing. Or you can examine your module's namespace and export each subroutine. (I don't care to search for how to do that at the moment, but Perl makes this fairly easy.) Or you could just stick your subroutines in the main package:

 package main;
 sub firstsub() { ... }

(I don't think that's a good idea, but you know better than I do what you are trying to accomplish.)

There's nothing wrong with doing this provided you know what you are doing and aren't just trying to avoid thinking about your interface to the outside world.

ysth , 2009-04-09 01:29:04

Perhaps you would be interested in one of the Export* modules on CPAN that lets you mark subs as exportable simply by adding an attribute to the sub definition? (Don't remember which one it was, though.)

echo , 2014-10-11 18:23:01

https://metacpan.org/pod/Exporter::Auto

Exporter::Auto. this is all you need.

Tero Niemi , 2013-04-02 00:32:25

Although it is not usually wise to dump all sub s from module into the caller namespace, it is sometimes useful (and more DRY!) to automatically generate @EXPORT_OK and %EXPORT_TAGS variables.

The easiest method is to extend the Exporter. A simple example is something like this:

package Exporter::AutoOkay;
#
#   Automatically add all subroutines from caller package into the
#   @EXPORT_OK array. In the package use like Exporter, f.ex.:
#
#       use parent 'Exporter::AutoOkay';
#
use warnings;
use strict;
no strict 'refs';

require Exporter;

sub import {
    my $package = $_[0].'::';

    # Get the list of exportable items
    my @export_ok = (@{$package.'EXPORT_OK'});

    # Automatically add all subroutines from package into the list
    foreach (keys %{$package}) {
        next unless defined &{$package.$_};
        push @export_ok, $_;
    }

    # Set variable ready for Exporter
    @{$package.'EXPORT_OK'} = @export_ok;

    # Let Exporter do the rest
    goto &Exporter::import;
}

1;

Note the use of goto that removes us from the caller stack.

A more complete example can be found here: http://pastebin.com/Z1QWzcpZ It automatically generates tag groups from subroutine prefixes.

Sérgio , 2013-11-14 21:38:06

case 1

Library is :

package mycommon;

use strict;
use warnings;

sub onefunctionthatyoumadeonlibary() {
}
1;

you can use it, calling common:: :

#!/usr/bin/perl
use strict;
use warnings;
use mycommon;

common::onefunctionthatyoumadeonlibary()
case 2

Library is , yousimple export them :

package mycommon;

use strict;
use warnings;

use base 'Exporter';

our @EXPORT = qw(onefunctionthatyoumadeonlibary);
sub onefunctionthatyoumadeonlibary() {
}
1;

use it in same "namespace":

#!/usr/bin/perl
use strict;
use warnings;
use mycommon qw(onefunctionthatyoumadeonlibary);

onefunctionthatyoumadeonlibary()

Also we can do a mix of this two cases , we can export more common functions to use it without calling the packages name and other functions that we only call it with package name and that ones don't need to be exported.

> ,

You will have to do some typeglob munging. I describe something similar here:

Is there a way to "use" a single file that in turn uses multiple others in Perl?

The import routine there should do exactly what you want -- just don't import any symbols into your own namespace.

Ville M ,

I would like to expose all subs into my namespace without having to list them one at a time:
@EXPORT = qw( firstsub secondsub third sub etc );

Using fully qualified names would require bunch of change to existing code so I'd rather not do that.

Is there @EXPORT_ALL?

I think documentation says it's a bad idea, but I'd like to do it anyway, or at least know how.

To answer Jon's why: right now for quick refactoring I want to move of bunch of subs into their own package with least hassle and code changes to the existing scripts (where those subs are currenty used and often repeated).

Also, mostly, I was just curious. (since it seemed like that Exporter might as well have that as standard feature, but somewhat surprisingly based on answers so far it doesn't)

brian d foy , 2009-04-08 23:58:35

Don't do any exporting at all, and don't declare a package name in your library. Just load the file with require and everything will be in the current package. Easy peasy.

Michael Carman , 2009-04-09 00:15:10

Don't. But if you really want to... write a custom import that walks the symbol table and export all the named subroutines.
# Export all subs in package. Not for use in production code!
sub import {
    no strict 'refs';

    my $caller = caller;

    while (my ($name, $symbol) = each %{__PACKAGE__ . '::'}) {
        next if      $name eq 'BEGIN';   # don't export BEGIN blocks
        next if      $name eq 'import';  # don't export this sub
        next unless *{$symbol}{CODE};    # export subs only

        my $imported = $caller . '::' . $name;
        *{ $imported } = \*{ $symbol };
    }
}

Chas. Owens ,

Warning, the code following is as bad an idea as exporting everything:
package Expo;

use base "Exporter";

seek DATA, 0, 0; #move DATA back to package

#read this file looking for sub names
our @EXPORT = map { /^sub\s+([^({\s]+)/ ? $1 : () } <DATA>;

my $sub = sub {}; #make sure anon funcs aren't grabbed

sub foo($) {
    print shift, "\n";
}

sub bar ($) {
    print shift, "\n";
}

sub baz{
    print shift,"\n";
}

sub quux {
    print shift,"\n";
}

1;

__DATA__

Here is the some code that uses the module:

#!/usr/bin/perl

use strict;
use warnings;

use Expo;

print map { "[$_]\n" } @Expo::EXPORT;

foo("foo");
bar("bar");
baz("baz");
quux("quux");

And here is its output:

[foo]
[bar]
[baz]
[quux]
foo
bar
baz
quux

Jon Ericson , 2009-04-08 22:33:36

You can always call subroutines in there fully-specified form:
MyModule::firstsub();

For modules I write internally, I find this convention works fairly well. It's a bit more typing, but tends to be better documentation.

Take a look at perldoc perlmod for more information about what you are trying to accomplish.

More generally, you could look at Exporter 's code and see how it uses glob aliasing. Or you can examine your module's namespace and export each subroutine. (I don't care to search for how to do that at the moment, but Perl makes this fairly easy.) Or you could just stick your subroutines in the main package:

 package main;
 sub firstsub() { ... }

(I don't think that's a good idea, but you know better than I do what you are trying to accomplish.)

There's nothing wrong with doing this provided you know what you are doing and aren't just trying to avoid thinking about your interface to the outside world.

ysth , 2009-04-09 01:29:04

Perhaps you would be interested in one of the Export* modules on CPAN that lets you mark subs as exportable simply by adding an attribute to the sub definition? (Don't remember which one it was, though.)

echo , 2014-10-11 18:23:01

https://metacpan.org/pod/Exporter::Auto

Exporter::Auto. this is all you need.

Tero Niemi , 2013-04-02 00:32:25

Although it is not usually wise to dump all sub s from module into the caller namespace, it is sometimes useful (and more DRY!) to automatically generate @EXPORT_OK and %EXPORT_TAGS variables.

The easiest method is to extend the Exporter. A simple example is something like this:

package Exporter::AutoOkay;
#
#   Automatically add all subroutines from caller package into the
#   @EXPORT_OK array. In the package use like Exporter, f.ex.:
#
#       use parent 'Exporter::AutoOkay';
#
use warnings;
use strict;
no strict 'refs';

require Exporter;

sub import {
    my $package = $_[0].'::';

    # Get the list of exportable items
    my @export_ok = (@{$package.'EXPORT_OK'});

    # Automatically add all subroutines from package into the list
    foreach (keys %{$package}) {
        next unless defined &{$package.$_};
        push @export_ok, $_;
    }

    # Set variable ready for Exporter
    @{$package.'EXPORT_OK'} = @export_ok;

    # Let Exporter do the rest
    goto &Exporter::import;
}

1;

Note the use of goto that removes us from the caller stack.

A more complete example can be found here: http://pastebin.com/Z1QWzcpZ It automatically generates tag groups from subroutine prefixes.

Sérgio , 2013-11-14 21:38:06

case 1

Library is :

package mycommon;

use strict;
use warnings;

sub onefunctionthatyoumadeonlibary() {
}
1;

you can use it, calling common:: :

#!/usr/bin/perl
use strict;
use warnings;
use mycommon;

common::onefunctionthatyoumadeonlibary()
case 2

Library is , yousimple export them :

package mycommon;

use strict;
use warnings;

use base 'Exporter';

our @EXPORT = qw(onefunctionthatyoumadeonlibary);
sub onefunctionthatyoumadeonlibary() {
}
1;

use it in same "namespace":

#!/usr/bin/perl
use strict;
use warnings;
use mycommon qw(onefunctionthatyoumadeonlibary);

onefunctionthatyoumadeonlibary()

Also we can do a mix of this two cases , we can export more common functions to use it without calling the packages name and other functions that we only call it with package name and that ones don't need to be exported.

> ,

You will have to do some typeglob munging. I describe something similar here:

Is there a way to "use" a single file that in turn uses multiple others in Perl?

The import routine there should do exactly what you want -- just don't import any symbols into your own namespace.

Ville M ,

I would like to expose all subs into my namespace without having to list them one at a time:
@EXPORT = qw( firstsub secondsub third sub etc );

Using fully qualified names would require bunch of change to existing code so I'd rather not do that.

Is there @EXPORT_ALL?

I think documentation says it's a bad idea, but I'd like to do it anyway, or at least know how.

To answer Jon's why: right now for quick refactoring I want to move of bunch of subs into their own package with least hassle and code changes to the existing scripts (where those subs are currenty used and often repeated).

Also, mostly, I was just curious. (since it seemed like that Exporter might as well have that as standard feature, but somewhat surprisingly based on answers so far it doesn't)

brian d foy , 2009-04-08 23:58:35

Don't do any exporting at all, and don't declare a package name in your library. Just load the file with require and everything will be in the current package. Easy peasy.

Michael Carman , 2009-04-09 00:15:10

Don't. But if you really want to... write a custom import that walks the symbol table and export all the named subroutines.
# Export all subs in package. Not for use in production code!
sub import {
    no strict 'refs';

    my $caller = caller;

    while (my ($name, $symbol) = each %{__PACKAGE__ . '::'}) {
        next if      $name eq 'BEGIN';   # don't export BEGIN blocks
        next if      $name eq 'import';  # don't export this sub
        next unless *{$symbol}{CODE};    # export subs only

        my $imported = $caller . '::' . $name;
        *{ $imported } = \*{ $symbol };
    }
}

Chas. Owens ,

Warning, the code following is as bad an idea as exporting everything:
package Expo;

use base "Exporter";

seek DATA, 0, 0; #move DATA back to package

#read this file looking for sub names
our @EXPORT = map { /^sub\s+([^({\s]+)/ ? $1 : () } <DATA>;

my $sub = sub {}; #make sure anon funcs aren't grabbed

sub foo($) {
    print shift, "\n";
}

sub bar ($) {
    print shift, "\n";
}

sub baz{
    print shift,"\n";
}

sub quux {
    print shift,"\n";
}

1;

__DATA__

Here is the some code that uses the module:

#!/usr/bin/perl

use strict;
use warnings;

use Expo;

print map { "[$_]\n" } @Expo::EXPORT;

foo("foo");
bar("bar");
baz("baz");
quux("quux");

And here is its output:

[foo]
[bar]
[baz]
[quux]
foo
bar
baz
quux

Jon Ericson , 2009-04-08 22:33:36

You can always call subroutines in there fully-specified form:
MyModule::firstsub();

For modules I write internally, I find this convention works fairly well. It's a bit more typing, but tends to be better documentation.

Take a look at perldoc perlmod for more information about what you are trying to accomplish.

More generally, you could look at Exporter 's code and see how it uses glob aliasing. Or you can examine your module's namespace and export each subroutine. (I don't care to search for how to do that at the moment, but Perl makes this fairly easy.) Or you could just stick your subroutines in the main package:

 package main;
 sub firstsub() { ... }

(I don't think that's a good idea, but you know better than I do what you are trying to accomplish.)

There's nothing wrong with doing this provided you know what you are doing and aren't just trying to avoid thinking about your interface to the outside world.

ysth , 2009-04-09 01:29:04

Perhaps you would be interested in one of the Export* modules on CPAN that lets you mark subs as exportable simply by adding an attribute to the sub definition? (Don't remember which one it was, though.)

echo , 2014-10-11 18:23:01

https://metacpan.org/pod/Exporter::Auto

Exporter::Auto. this is all you need.

Tero Niemi , 2013-04-02 00:32:25

Although it is not usually wise to dump all sub s from module into the caller namespace, it is sometimes useful (and more DRY!) to automatically generate @EXPORT_OK and %EXPORT_TAGS variables.

The easiest method is to extend the Exporter. A simple example is something like this:

package Exporter::AutoOkay;
#
#   Automatically add all subroutines from caller package into the
#   @EXPORT_OK array. In the package use like Exporter, f.ex.:
#
#       use parent 'Exporter::AutoOkay';
#
use warnings;
use strict;
no strict 'refs';

require Exporter;

sub import {
    my $package = $_[0].'::';

    # Get the list of exportable items
    my @export_ok = (@{$package.'EXPORT_OK'});

    # Automatically add all subroutines from package into the list
    foreach (keys %{$package}) {
        next unless defined &{$package.$_};
        push @export_ok, $_;
    }

    # Set variable ready for Exporter
    @{$package.'EXPORT_OK'} = @export_ok;

    # Let Exporter do the rest
    goto &Exporter::import;
}

1;

Note the use of goto that removes us from the caller stack.

A more complete example can be found here: http://pastebin.com/Z1QWzcpZ It automatically generates tag groups from subroutine prefixes.

Sérgio , 2013-11-14 21:38:06

case 1

Library is :

package mycommon;

use strict;
use warnings;

sub onefunctionthatyoumadeonlibary() {
}
1;

you can use it, calling common:: :

#!/usr/bin/perl
use strict;
use warnings;
use mycommon;

common::onefunctionthatyoumadeonlibary()
case 2

Library is , yousimple export them :

package mycommon;

use strict;
use warnings;

use base 'Exporter';

our @EXPORT = qw(onefunctionthatyoumadeonlibary);
sub onefunctionthatyoumadeonlibary() {
}
1;

use it in same "namespace":

#!/usr/bin/perl
use strict;
use warnings;
use mycommon qw(onefunctionthatyoumadeonlibary);

onefunctionthatyoumadeonlibary()

Also we can do a mix of this two cases , we can export more common functions to use it without calling the packages name and other functions that we only call it with package name and that ones don't need to be exported.

> ,

You will have to do some typeglob munging. I describe something similar here:

Is there a way to "use" a single file that in turn uses multiple others in Perl?

The import routine there should do exactly what you want -- just don't import any symbols into your own namespace.

[Dec 01, 2019] function - How can I export all subs in a Perl package - Stack Overflow

Jan 01, 2009 | stackoverflow.com

How can I export all subs in a Perl package? Ask Question Asked 10 years, 7 months ago Active 3 years, 5 months ago Viewed 18k times


Ville M ,

I would like to expose all subs into my namespace without having to list them one at a time:
@EXPORT = qw( firstsub secondsub third sub etc );

Using fully qualified names would require bunch of change to existing code so I'd rather not do that.

Is there @EXPORT_ALL?

I think documentation says it's a bad idea, but I'd like to do it anyway, or at least know how.

To answer Jon's why: right now for quick refactoring I want to move of bunch of subs into their own package with least hassle and code changes to the existing scripts (where those subs are currenty used and often repeated).

Also, mostly, I was just curious. (since it seemed like that Exporter might as well have that as standard feature, but somewhat surprisingly based on answers so far it doesn't)

brian d foy , 2009-04-08 23:58:35

Don't do any exporting at all, and don't declare a package name in your library. Just load the file with require and everything will be in the current package. Easy peasy.

Michael Carman , 2009-04-09 00:15:10

Don't. But if you really want to... write a custom import that walks the symbol table and export all the named subroutines.
# Export all subs in package. Not for use in production code!
sub import {
    no strict 'refs';

    my $caller = caller;

    while (my ($name, $symbol) = each %{__PACKAGE__ . '::'}) {
        next if      $name eq 'BEGIN';   # don't export BEGIN blocks
        next if      $name eq 'import';  # don't export this sub
        next unless *{$symbol}{CODE};    # export subs only

        my $imported = $caller . '::' . $name;
        *{ $imported } = \*{ $symbol };
    }
}

Chas. Owens ,

Warning, the code following is as bad an idea as exporting everything:
package Expo;

use base "Exporter";

seek DATA, 0, 0; #move DATA back to package

#read this file looking for sub names
our @EXPORT = map { /^sub\s+([^({\s]+)/ ? $1 : () } <DATA>;

my $sub = sub {}; #make sure anon funcs aren't grabbed

sub foo($) {
    print shift, "\n";
}

sub bar ($) {
    print shift, "\n";
}

sub baz{
    print shift,"\n";
}

sub quux {
    print shift,"\n";
}

1;

__DATA__

Here is the some code that uses the module:

#!/usr/bin/perl

use strict;
use warnings;

use Expo;

print map { "[$_]\n" } @Expo::EXPORT;

foo("foo");
bar("bar");
baz("baz");
quux("quux");

And here is its output:

[foo]
[bar]
[baz]
[quux]
foo
bar
baz
quux

Jon Ericson , 2009-04-08 22:33:36

You can always call subroutines in there fully-specified form:
MyModule::firstsub();

For modules I write internally, I find this convention works fairly well. It's a bit more typing, but tends to be better documentation.

Take a look at perldoc perlmod for more information about what you are trying to accomplish.

More generally, you could look at Exporter 's code and see how it uses glob aliasing. Or you can examine your module's namespace and export each subroutine. (I don't care to search for how to do that at the moment, but Perl makes this fairly easy.) Or you could just stick your subroutines in the main package:

 package main;
 sub firstsub() { ... }

(I don't think that's a good idea, but you know better than I do what you are trying to accomplish.)

There's nothing wrong with doing this provided you know what you are doing and aren't just trying to avoid thinking about your interface to the outside world.

ysth , 2009-04-09 01:29:04

Perhaps you would be interested in one of the Export* modules on CPAN that lets you mark subs as exportable simply by adding an attribute to the sub definition? (Don't remember which one it was, though.)

echo , 2014-10-11 18:23:01

https://metacpan.org/pod/Exporter::Auto

Exporter::Auto. this is all you need.

Tero Niemi , 2013-04-02 00:32:25

Although it is not usually wise to dump all sub s from module into the caller namespace, it is sometimes useful (and more DRY!) to automatically generate @EXPORT_OK and %EXPORT_TAGS variables.

The easiest method is to extend the Exporter. A simple example is something like this:

package Exporter::AutoOkay;
#
#   Automatically add all subroutines from caller package into the
#   @EXPORT_OK array. In the package use like Exporter, f.ex.:
#
#       use parent 'Exporter::AutoOkay';
#
use warnings;
use strict;
no strict 'refs';

require Exporter;

sub import {
    my $package = $_[0].'::';

    # Get the list of exportable items
    my @export_ok = (@{$package.'EXPORT_OK'});

    # Automatically add all subroutines from package into the list
    foreach (keys %{$package}) {
        next unless defined &{$package.$_};
        push @export_ok, $_;
    }

    # Set variable ready for Exporter
    @{$package.'EXPORT_OK'} = @export_ok;

    # Let Exporter do the rest
    goto &Exporter::import;
}

1;

Note the use of goto that removes us from the caller stack.

A more complete example can be found here: http://pastebin.com/Z1QWzcpZ It automatically generates tag groups from subroutine prefixes.

Sérgio , 2013-11-14 21:38:06

case 1

Library is :

package mycommon;

use strict;
use warnings;

sub onefunctionthatyoumadeonlibary() {
}
1;

you can use it, calling common:: :

#!/usr/bin/perl
use strict;
use warnings;
use mycommon;

common::onefunctionthatyoumadeonlibary()
case 2

Library is , yousimple export them :

package mycommon;

use strict;
use warnings;

use base 'Exporter';

our @EXPORT = qw(onefunctionthatyoumadeonlibary);
sub onefunctionthatyoumadeonlibary() {
}
1;

use it in same "namespace":

#!/usr/bin/perl
use strict;
use warnings;
use mycommon qw(onefunctionthatyoumadeonlibary);

onefunctionthatyoumadeonlibary()

Also we can do a mix of this two cases , we can export more common functions to use it without calling the packages name and other functions that we only call it with package name and that ones don't need to be exported.

> ,

You will have to do some typeglob munging. I describe something similar here:

Is there a way to "use" a single file that in turn uses multiple others in Perl?

The import routine there should do exactly what you want -- just don't import any symbols into your own namespace.

[Nov 23, 2019] What is the meaning of a single and a double underscore before an object name?

Notable quotes:
"... does not import objects whose name starts with an underscore. ..."
Aug 19, 2009 | stackoverflow.com
Can someone please explain the exact meaning of having leading underscores before an object's name in Python? Also, explain the difference between a single and a double leading underscore. Also, does that meaning stay the same whether the object in question is a variable, a function, a method, etc.?

Andrew Keeton , 2009-08-19 17:15:53

Single Underscore

Names, in a class, with a leading underscore are simply to indicate to other programmers that the attribute or method is intended to be private. However, nothing special is done with the name itself.

To quote PEP-8 :

_single_leading_underscore: weak "internal use" indicator. E.g. from M import * does not import objects whose name starts with an underscore.

Double Underscore (Name Mangling)

From the Python docs :

Any identifier of the form __spam (at least two leading underscores, at most one trailing underscore) is textually replaced with _classname__spam , where classname is the current class name with leading underscore(s) stripped. This mangling is done without regard to the syntactic position of the identifier, so it can be used to define class-private instance and class variables, methods, variables stored in globals, and even variables stored in instances. private to this class on instances of other classes.

And a warning from the same page:

Name mangling is intended to give classes an easy way to define "private" instance variables and methods, without having to worry about instance variables defined by derived classes, or mucking with instance variables by code outside the class. Note that the mangling rules are designed mostly to avoid accidents; it still is possible for a determined soul to access or modify a variable that is considered private.

Example
>>> class MyClass():
...     def __init__(self):
...             self.__superprivate = "Hello"
...             self._semiprivate = ", world!"
...
>>> mc = MyClass()
>>> print mc.__superprivate
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
AttributeError: myClass instance has no attribute '__superprivate'
>>> print mc._semiprivate
, world!
>>> print mc.__dict__
{'_MyClass__superprivate': 'Hello', '_semiprivate': ', world!'}

Alex Martelli , 2009-08-19 17:52:36

Excellent answers so far but some tidbits are missing. A single leading underscore isn't exactly just a convention: if you use from foobar import * , and module foobar does not define an __all__ list, the names imported from the module do not include those with a leading underscore. Let's say it's mostly a convention, since this case is a pretty obscure corner;-).

The leading-underscore convention is widely used not just for private names, but also for what C++ would call protected ones -- for example, names of methods that are fully intended to be overridden by subclasses (even ones that have to be overridden since in the base class they raise NotImplementedError !-) are often single-leading-underscore names to indicate to code using instances of that class (or subclasses) that said methods are not meant to be called directly.

For example, to make a thread-safe queue with a different queueing discipline than FIFO, one imports Queue, subclasses Queue.Queue, and overrides such methods as _get and _put ; "client code" never calls those ("hook") methods, but rather the ("organizing") public methods such as put and get (this is known as the Template Method design pattern -- see e.g. here for an interesting presentation based on a video of a talk of mine on the subject, with the addition of synopses of the transcript).

Ned Batchelder , 2009-08-19 17:21:29

__foo__ : this is just a convention, a way for the Python system to use names that won't conflict with user names.

_foo : this is just a convention, a way for the programmer to indicate that the variable is private (whatever that means in Python).

__foo : this has real meaning: the interpreter replaces this name with _classname__foo as a way to ensure that the name will not overlap with a similar name in another class.

No other form of underscores have meaning in the Python world.

There's no difference between class, variable, global, etc in these conventions.

2 revs, 2 users 93%
, 2016-05-17 10:09:08

._variable is semiprivate and meant just for convention

.__variable is often incorrectly considered superprivate, while it's actual meaning is just to namemangle to prevent accidental access [1]

.__variable__ is typically reserved for builtin methods or variables

You can still access .__mangled variables if you desperately want to. The double underscores just namemangles, or renames, the variable to something like instance._className__mangled

Example:

class Test(object):
    def __init__(self):
        self.__a = 'a'
        self._b = 'b'

>>> t = Test()
>>> t._b
'b'

t._b is accessible because it is only hidden by convention

>>> t.__a
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
AttributeError: 'Test' object has no attribute '__a'

t.__a isn't found because it no longer exists due to namemangling

>>> t._Test__a
'a'

By accessing instance._className__variable instead of just the double underscore name, you can access the hidden value

9 revs, 8 users 82%
, 2018-08-21 19:42:09

Single underscore at the beginning:

Python doesn't have real private methods. Instead, one underscore at the start of a method or attribute name means you shouldn't access this method, because it's not part of the API.

class BaseForm(StrAndUnicode):

    def _get_errors(self):
        "Returns an ErrorDict for the data provided for the form"
        if self._errors is None:
            self.full_clean()
        return self._errors

    errors = property(_get_errors)

(This code snippet was taken from django source code: django/forms/forms.py). In this code, errors is a public property, but the method this property calls, _get_errors, is "private", so you shouldn't access it.

Two underscores at the beginning:

This causes a lot of confusion. It should not be used to create a private method. It should be used to avoid your method being overridden by a subclass or accessed accidentally. Let's see an example:

class A(object):
    def __test(self):
        print "I'm a test method in class A"

    def test(self):
        self.__test()

a = A()
a.test()
# a.__test() # This fails with an AttributeError
a._A__test() # Works! We can access the mangled name directly!

Output:

$ python test.py
I'm test method in class A
I'm test method in class A

Now create a subclass B and do customization for __test method

class B(A):
    def __test(self):
        print "I'm test method in class B"

b = B()
b.test()

Output will be....

$ python test.py
I'm test method in class A

As we have seen, A.test() didn't call B.__test() methods, as we might expect. But in fact, this is the correct behavior for __. The two methods called __test() are automatically renamed (mangled) to _A__test() and _B__test(), so they do not accidentally override. When you create a method starting with __ it means that you don't want to anyone to be able to override it, and you only intend to access it from inside its own class.

Two underscores at the beginning and at the end:

When we see a method like __this__ , don't call it. This is a method which python is meant to call, not you. Let's take a look:

>>> name = "test string"
>>> name.__len__()
11
>>> len(name)
11

>>> number = 10
>>> number.__add__(40)
50
>>> number + 50
60

There is always an operator or native function which calls these magic methods. Sometimes it's just a hook python calls in specific situations. For example __init__() is called when the object is created after __new__() is called to build the instance...

Let's take an example...

class FalseCalculator(object):

    def __init__(self, number):
        self.number = number

    def __add__(self, number):
        return self.number - number

    def __sub__(self, number):
        return self.number + number

number = FalseCalculator(20)
print number + 10      # 10
print number - 20      # 40

For more details, see the PEP-8 guide . For more magic methods, see this PDF .

Tim D , 2012-01-11 16:28:22

Sometimes you have what appears to be a tuple with a leading underscore as in
def foo(bar):
    return _('my_' + bar)

In this case, what's going on is that _() is an alias for a localization function that operates on text to put it into the proper language, etc. based on the locale. For example, Sphinx does this, and you'll find among the imports

from sphinx.locale import l_, _

and in sphinx.locale, _() is assigned as an alias of some localization function.

Dev Maha , 2013-04-15 01:58:14

If one really wants to make a variable read-only, IMHO the best way would be to use property() with only getter passed to it. With property() we can have complete control over the data.
class PrivateVarC(object):

    def get_x(self):
        pass

    def set_x(self, val):
        pass

    rwvar = property(get_p, set_p)  

    ronly = property(get_p)

I understand that OP asked a little different question but since I found another question asking for 'how to set private variables' marked duplicate with this one, I thought of adding this additional info here.

SilentGhost ,

Single leading underscores is a convention. there is no difference from the interpreter's point of view if whether names starts with a single underscore or not.

Double leading and trailing underscores are used for built-in methods, such as __init__ , __bool__ , etc.

Double leading underscores w/o trailing counterparts are a convention too, however, the class methods will be mangled by the interpreter. For variables or basic function names no difference exists.

3 revs
, 2018-12-16 11:41:34

Since so many people are referring to Raymond's talk , I'll just make it a little easier by writing down what he said:

The intention of the double underscores was not about privacy. The intention was to use it exactly like this

class Circle(object):

    def __init__(self, radius):
        self.radius = radius

    def area(self):
        p = self.__perimeter()
        r = p / math.pi / 2.0
        return math.pi * r ** 2.0

    def perimeter(self):
        return 2.0 * math.pi * self.radius

    __perimeter = perimeter  # local reference


class Tire(Circle):

    def perimeter(self):
        return Circle.perimeter(self) * 1.25

It's actually the opposite of privacy, it's all about freedom. It makes your subclasses free to override any one method without breaking the others .

Say you don't keep a local reference of perimeter in Circle . Now, a derived class Tire overrides the implementation of perimeter , without touching area . When you call Tire(5).area() , in theory it should still be using Circle.perimeter for computation, but in reality it's using Tire.perimeter , which is not the intended behavior. That's why we need a local reference in Circle.

But why __perimeter instead of _perimeter ? Because _perimeter still gives derived class the chance to override:

class Tire(Circle):

    def perimeter(self):
        return Circle.perimeter(self) * 1.25

    _perimeter = perimeter

Double underscores has name mangling, so there's a very little chance that the local reference in parent class get override in derived class. thus " makes your subclasses free to override any one method without breaking the others ".

If your class won't be inherited, or method overriding does not break anything, then you simply don't need __double_leading_underscore .

u0b34a0f6ae , 2009-08-19 17:31:04

Your question is good, it is not only about methods. Functions and objects in modules are commonly prefixed with one underscore as well, and can be prefixed by two.

But __double_underscore names are not name-mangled in modules, for example. What happens is that names beginning with one (or more) underscores are not imported if you import all from a module (from module import *), nor are the names shown in help(module).

Marc , 2014-08-22 19:15:48

Here is a simple illustrative example on how double underscore properties can affect an inherited class. So with the following setup:
class parent(object):
    __default = "parent"
    def __init__(self, name=None):
        self.default = name or self.__default

    @property
    def default(self):
        return self.__default

    @default.setter
    def default(self, value):
        self.__default = value


class child(parent):
    __default = "child"

if you then create a child instance in the python REPL, you will see the below

child_a = child()
child_a.default            # 'parent'
child_a._child__default    # 'child'
child_a._parent__default   # 'parent'

child_b = child("orphan")
## this will show 
child_b.default            # 'orphan'
child_a._child__default    # 'child'
child_a._parent__default   # 'orphan'

This may be obvious to some, but it caught me off guard in a much more complex environment

aptro , 2015-02-07 17:57:10

"Private" instance variables that cannot be accessed except from inside an object don't exist in Python. However, there is a convention that is followed by most Python code: a name prefixed with an underscore (e.g. _spam) should be treated as a non-public part of the API (whether it is a function, a method or a data member). It should be considered an implementation detail and subject to change without notice.

reference https://docs.python.org/2/tutorial/classes.html#private-variables-and-class-local-references

grepit , 2019-01-21 22:23:39

Great answers and all are correct.I have provided simple example along with simple definition/meaning.

Meaning:

some_variable --► it's public anyone can see this.

_some_variable --► it's public anyone can see this but it's a convention to indicate private... warning no enforcement is done by Python.

__some_varaible --► Python replaces the variable name with _classname__some_varaible (AKA name mangling) and it reduces/hides it's visibility and be more like private variable.

Just to be honest here According to Python documentation

""Private" instance variables that cannot be accessed except from inside an object don't exist in Python"

The example:

class A():
    here="abc"
    _here="_abc"
    __here="__abc"


aObject=A()
print(aObject.here) 
print(aObject._here)
# now if we try to print __here then it will fail because it's not public variable 
#print(aObject.__here)

2 revs
, 2017-11-04 17:51:49

Getting the facts of _ and __ is pretty easy; the other answers express them pretty well. The usage is much harder to determine.

This is how I see it:

_

Should be used to indicate that a function is not for public use as for example an API. This and the import restriction make it behave much like internal in c#.

__

Should be used to avoid name collision in the inheritace hirarchy and to avoid latebinding. Much like private in c#.

==>

If you want to indicate that something is not for public use, but it should act like protected use _ . If you want to indicate that something is not for public use, but it should act like private use __ .

This is also a quote that I like very much:

The problem is that the author of a class may legitimately think "this attribute/method name should be private, only accessible from within this class definition" and use the __private convention. But later on, a user of that class may make a subclass that legitimately needs access to that name. So either the superclass has to be modified (which may be difficult or impossible), or the subclass code has to use manually mangled names (which is ugly and fragile at best).

But the problem with that is in my opinion that if there's no IDE that warns you when you override methods, finding the error might take you a while if you have accidentially overriden a method from a base-class.

[Nov 23, 2019] Static local variables in Perl

Jan 01, 2012 | stackoverflow.com

Ask Question Asked 7 years, 5 months ago Active 2 years, 8 months ago Viewed 12k times


Charles , 2012-05-31 20:50:19

I'm looking for advice on Perl best practices. I wrote a script which had a complicated regular expression:
my $regex = qr/complicated/;

# ...

sub foo {
  # ...

  if (/$regex/)
  # ...
}

where foo is a function which is called often, and $regex is not used outside that function. What is the best way to handle situations like this? I only want it to be interpreted once, since it's long and complicated. But it seems a bit questionable to have it in global scope since it's only used in that sub. Is there a reasonable way to declare it static?

A similar issue arises with another possibly-unjustified global. It reads in the current date and time and formats it appropriately. This is also used many times, and again only in one function. But in this case it's even more important that it not be re-initialized, since I want all instances of the date-time to be the same from a given invocation of the script, even if the minutes roll over during execution.

At the moment I have something like

my ($regex, $DT);

sub driver {
  $regex = qr/complicated/;
  $DT = dateTime();
  # ...
}

# ...

driver();

which at least slightly segregates it. But perhaps there are better ways.

Again: I'm looking for the right way to do this, in terms of following best practices and Perl idioms. Performance is nice but readability and other needs take priority if I can't have everything.

hobbs ,

If you're using perl 5.10+, use a state variable.
use feature 'state';
# use 5.010; also works

sub womble {
    state $foo = something_expensive();
    return $foo ** 2;
}

will only call something_expensive once.

If you need to work with older perls, then use a lexical variable in an outer scope with an extra pair of braces:

{
    my $foo = something_expensive();
    sub womble {
        return $foo ** 2;
    }
}

this keeps $foo from leaking to anyone except for womble .

ikegami , 2012-05-31 21:14:04

Is there any interpolation in the pattern? If not, the pattern will only be compiled once no matter how many times the qr// is executed.
$ perl -Mre=debug -e'qr/foo/ for 1..10' 2>&1 | grep Compiling | wc -l
1

$ perl -Mre=debug -e'qr/foo$_/ for 1..10' 2>&1 | grep Compiling | wc -l
10

Even if there is interpolation, the pattern will only be compiled if the interpolated variables have changed.

$ perl -Mre=debug -e'$x=123; qr/foo$x/ for 1..10;' 2>&1 | grep Compiling | wc -l
1

$ perl -Mre=debug -e'qr/foo$_/ for 1..10' 2>&1 | grep Compiling | wc -l
10

Otherwise, you can use

{
   my $re = qr/.../;
   sub foo {
      ...
      /$re/
      ...
   }
}

or

use feature qw( state );
sub foo {
   state $re = qr/.../;
   ...
   /$re/
   ...
}

Alan Rocker , 2014-07-02 16:25:27

Regexes can be specified with the "o" modifier, which says "compile pattern once only" - in the 3rd. edition of the Camel, see p. 147

zoul ,

There's a state keyword that might be a good fit for this situation:
sub foo {
    state $regex = /.../;
    ...
}

TrueY , 2015-01-23 10:14:12

I would like to complete ikegami 's great answer. Some more words I would like to waste on the definition of local variables in pre 5.10 perl .

Let's see a simple example code:

#!/bin/env perl 

use strict;
use warnings;

{ # local 
my $local = "After Crying";
sub show { print $local,"\n"; }
} # local

sub show2;

show;
show2;

exit;

{ # local 
my $local = "Solaris";
sub show2 { print $local,"\n"; }
} # local

The user would expect that both sub will print the local variable, but this is not true!

Output:

After Crying
Use of uninitialized value $local in print at ./x.pl line 20.

The reason is that show2 is parsed, but the initialization of the local variable is not executed! (Of course if exit is removed and a show2 is added at the end, Solaris will be printed in the thirds line)

This can be fixed easily:

{ # local 
my $local;
BEGIN { $local = "Solaris"; }
sub show2 { print $local,"\n"; }
} # local

And now the output what was expected:

After Crying
Solaris

But state in 5.10+ is a better choice...

I hope this helps!

[Nov 23, 2019] How can I remove a trailing newline?

Jan 01, 2008 | stackoverflow.com

Ask Question Asked 11 years ago Active 16 days ago Viewed 1.7m times


, 2008-11-08 18:25:24

What is the Python equivalent of Perl's chomp function, which removes the last character of a string if it is a newline?

9 revs, 7 users 37%
, 2017-05-11 19:54:59

Try the method rstrip() (see doc Python 2 and Python 3 )
>>> 'test string\n'.rstrip()
'test string'

Python's rstrip() method strips all kinds of trailing whitespace by default, not just one newline as Perl does with chomp .

>>> 'test string \n \r\n\n\r \n\n'.rstrip()
'test string'

To strip only newlines:

>>> 'test string \n \r\n\n\r \n\n'.rstrip('\n')
'test string \n \r\n\n\r '

There are also the methods lstrip() and strip() :

>>> s = "   \n\r\n  \n  abc   def \n\r\n  \n  "
>>> s.strip()
'abc   def'
>>> s.lstrip()
'abc   def \n\r\n  \n  '
>>> s.rstrip()
'   \n\r\n  \n  abc   def'

Ryan Ginstrom , 2008-11-09 05:52:43

And I would say the "pythonic" way to get lines without trailing newline characters is splitlines().
>>> text = "line 1\nline 2\r\nline 3\nline 4"
>>> text.splitlines()
['line 1', 'line 2', 'line 3', 'line 4']

Mike ,

The canonical way to strip end-of-line (EOL) characters is to use the string rstrip() method removing any trailing \r or \n. Here are examples for Mac, Windows, and Unix EOL characters.
>>> 'Mac EOL\r'.rstrip('\r\n')
'Mac EOL'
>>> 'Windows EOL\r\n'.rstrip('\r\n')
'Windows EOL'
>>> 'Unix EOL\n'.rstrip('\r\n')
'Unix EOL'

Using '\r\n' as the parameter to rstrip means that it will strip out any trailing combination of '\r' or '\n'. That's why it works in all three cases above.

This nuance matters in rare cases. For example, I once had to process a text file which contained an HL7 message. The HL7 standard requires a trailing '\r' as its EOL character. The Windows machine on which I was using this message had appended its own '\r\n' EOL character. Therefore, the end of each line looked like '\r\r\n'. Using rstrip('\r\n') would have taken off the entire '\r\r\n' which is not what I wanted. In that case, I simply sliced off the last two characters instead.

Note that unlike Perl's chomp function, this will strip all specified characters at the end of the string, not just one:

>>> "Hello\n\n\n".rstrip("\n")
"Hello"

, 2008-11-28 17:31:34

Note that rstrip doesn't act exactly like Perl's chomp() because it doesn't modify the string. That is, in Perl:
$x="a\n";

chomp $x

results in $x being "a" .

but in Python:

x="a\n"

x.rstrip()

will mean that the value of x is still "a\n" . Even x=x.rstrip() doesn't always give the same result, as it strips all whitespace from the end of the string, not just one newline at most.

Jamie ,

I might use something like this:
import os
s = s.rstrip(os.linesep)

I think the problem with rstrip("\n") is that you'll probably want to make sure the line separator is portable. (some antiquated systems are rumored to use "\r\n" ). The other gotcha is that rstrip will strip out repeated whitespace. Hopefully os.linesep will contain the right characters. the above works for me.

kiriloff , 2013-05-13 16:41:22

You may use line = line.rstrip('\n') . This will strip all newlines from the end of the string, not just one.

slec , 2015-03-09 08:02:55

s = s.rstrip()

will remove all newlines at the end of the string s . The assignment is needed because rstrip returns a new string instead of modifying the original string.

Alien Life Form ,

This would replicate exactly perl's chomp (minus behavior on arrays) for "\n" line terminator:
def chomp(x):
    if x.endswith("\r\n"): return x[:-2]
    if x.endswith("\n") or x.endswith("\r"): return x[:-1]
    return x

(Note: it does not modify string 'in place'; it does not strip extra trailing whitespace; takes \r\n in account)

Hackaholic ,

you can use strip:
line = line.strip()

demo:

>>> "\n\n hello world \n\n".strip()
'hello world'

mihaicc ,

"line 1\nline 2\r\n...".replace('\n', '').replace('\r', '')
>>> 'line 1line 2...'

or you could always get geekier with regexps :)

have fun!

Carlos Valiente , 2011-04-27 11:43:20

Careful with "foo".rstrip(os.linesep) : That will only chomp the newline characters for the platform where your Python is being executed. Imagine you're chimping the lines of a Windows file under Linux, for instance:
$ python
Python 2.7.1 (r271:86832, Mar 18 2011, 09:09:48) 
[GCC 4.5.0 20100604 [gcc-4_5-branch revision 160292]] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> import os, sys
>>> sys.platform
'linux2'
>>> "foo\r\n".rstrip(os.linesep)
'foo\r'
>>>

Use "foo".rstrip("\r\n") instead, as Mike says above.

minopret , 2013-10-23 01:32:11

An example in Python's documentation simply uses line.strip() .

Perl's chomp function removes one linebreak sequence from the end of a string only if it's actually there.

Here is how I plan to do that in Python, if process is conceptually the function that I need in order to do something useful to each line from this file:

import os
sep_pos = -len(os.linesep)
with open("file.txt") as f:
    for line in f:
        if line[sep_pos:] == os.linesep:
            line = line[:sep_pos]
        process(line)

ingydotnet ,

rstrip doesn't do the same thing as chomp, on so many levels. Read http://perldoc.perl.org/functions/chomp.html and see that chomp is very complex indeed.

However, my main point is that chomp removes at most 1 line ending, whereas rstrip will remove as many as it can.

Here you can see rstrip removing all the newlines:

>>> 'foo\n\n'.rstrip(os.linesep)
'foo'

A much closer approximation of typical Perl chomp usage can be accomplished with re.sub, like this:

>>> re.sub(os.linesep + r'\Z','','foo\n\n')
'foo\n'

Andrew Grimm ,

I don't program in Python, but I came across an FAQ at python.org advocating S.rstrip("\r\n") for python 2.2 or later.

, 2014-01-20 19:07:03

import re

r_unwanted = re.compile("[\n\t\r]")
r_unwanted.sub("", your_text)

Leozj ,

If your question is to clean up all the line breaks in a multiple line str object (oldstr), you can split it into a list according to the delimiter '\n' and then join this list into a new str(newstr).

newstr = "".join(oldstr.split('\n'))

kuzzooroo ,

I find it convenient to have be able to get the chomped lines via in iterator, parallel to the way you can get the un-chomped lines from a file object. You can do so with the following code:
def chomped_lines(it):
    return map(operator.methodcaller('rstrip', '\r\n'), it)

Sample usage:

with open("file.txt") as infile:
    for line in chomped_lines(infile):
        process(line)

Chij , 2011-11-30 14:04:19

workaround solution for special case:

if the newline character is the last character (as is the case with most file inputs), then for any element in the collection you can index as follows:

foobar= foobar[:-1]

to slice out your newline character.

user3780389 , 2017-04-26 17:58:16

It looks like there is not a perfect analog for perl's chomp . In particular, rstrip cannot handle multi-character newline delimiters like \r\n . However, splitlines does as pointed out here . Following my answer on a different question, you can combine join and splitlines to remove/replace all newlines from a string s :
''.join(s.splitlines())

The following removes exactly one trailing newline (as chomp would, I believe). Passing True as the keepends argument to splitlines retain the delimiters. Then, splitlines is called again to remove the delimiters on just the last "line":

def chomp(s):
    if len(s):
        lines = s.splitlines(True)
        last = lines.pop()
        return ''.join(lines + last.splitlines())
    else:
        return ''

Taylor Edmiston ,

I'm bubbling up my regular expression based answer from one I posted earlier in the comments of another answer. I think using re is a clearer more explicit solution to this problem than str.rstrip .
>>> import re

If you want to remove one or more trailing newline chars:

>>> re.sub(r'[\n\r]+$', '', '\nx\r\n')
'\nx'

If you want to remove newline chars everywhere (not just trailing):

>>> re.sub(r'[\n\r]+', '', '\nx\r\n')
'x'

If you want to remove only 1-2 trailing newline chars (i.e., \r , \n , \r\n , \n\r , \r\r , \n\n )

>>> re.sub(r'[\n\r]{1,2}$', '', '\nx\r\n\r\n')
'\nx\r'
>>> re.sub(r'[\n\r]{1,2}$', '', '\nx\r\n\r')
'\nx\r'
>>> re.sub(r'[\n\r]{1,2}$', '', '\nx\r\n')
'\nx'

I have a feeling what most people really want here, is to remove just one occurrence of a trailing newline character, either \r\n or \n and nothing more.

>>> re.sub(r'(?:\r\n|\n)$', '', '\nx\n\n', count=1)
'\nx\n'
>>> re.sub(r'(?:\r\n|\n)$', '', '\nx\r\n\r\n', count=1)
'\nx\r\n'
>>> re.sub(r'(?:\r\n|\n)$', '', '\nx\r\n', count=1)
'\nx'
>>> re.sub(r'(?:\r\n|\n)$', '', '\nx\n', count=1)
'\nx'

(The ?: is to create a non-capturing group.)

(By the way this is not what '...'.rstrip('\n', '').rstrip('\r', '') does which may not be clear to others stumbling upon this thread. str.rstrip strips as many of the trailing characters as possible, so a string like foo\n\n\n would result in a false positive of foo whereas you may have wanted to preserve the other newlines after stripping a single trailing one.)

Help me , 2016-05-20 12:29:21

Just use :
line = line.rstrip("\n")

or

line = line.strip("\n")

You don't need any of this complicated stuff

, 2016-11-22 18:30:37

>>> '   spacious   '.rstrip()
'   spacious'
>>> "AABAA".rstrip("A")
  'AAB'
>>> "ABBA".rstrip("AB") # both AB and BA are stripped
   ''
>>> "ABCABBA".rstrip("AB")
   'ABC'

internetional , 2016-11-22 20:17:58

There are three types of line endings that we normally encounter: \n , \r and \r\n . A rather simple regular expression in re.sub , namely r"\r?\n?$" , is able to catch them all.

(And we gotta catch 'em all , am I right?)

import re

re.sub(r"\r?\n?$", "", the_text, 1)

With the last argument, we limit the number of occurences replaced to one, mimicking chomp to some extent. Example:

import re

text_1 = "hellothere\n\n\n"
text_2 = "hellothere\n\n\r"
text_3 = "hellothere\n\n\r\n"

a = re.sub(r"\r?\n?$", "", text_1, 1)
b = re.sub(r"\r?\n?$", "", text_2, 1)
c = re.sub(r"\r?\n?$", "", text_3, 1)

... where a == b == c is True .

Venfah Nazir , 2018-06-15 07:24:21


This will work both for windows and linux (bit expensive with re sub if you are looking for only re solution)

import re 
if re.search("(\\r|)\\n$", line):
    line = re.sub("(\\r|)\\n$", "", line)

Stephen Miller ,

If you are concerned about speed (say you have a looong list of strings) and you know the nature of the newline char, string slicing is actually faster than rstrip. A little test to illustrate this:
import time

loops = 50000000

def method1(loops=loops):
    test_string = 'num\n'
    t0 = time.time()
    for num in xrange(loops):
        out_sting = test_string[:-1]
    t1 = time.time()
    print('Method 1: ' + str(t1 - t0))

def method2(loops=loops):
    test_string = 'num\n'
    t0 = time.time()
    for num in xrange(loops):
        out_sting = test_string.rstrip()
    t1 = time.time()
    print('Method 2: ' + str(t1 - t0))

method1()
method2()

Output:

Method 1: 3.92700004578
Method 2: 6.73000001907

sim , 2019-10-22 07:43:27

s = '''Hello  World \t\n\r\tHi There'''
# import the module string   
import string
# use the method translate to convert 
s.translate({ord(c): None for c in string.whitespace}
>>'HelloWorldHiThere'

With regex

s = '''  Hello  World 
\t\n\r\tHi '''
print(re.sub(r"\s+", "", s), sep='')  # \s matches all white spaces
>HelloWorldHi

Replace \n,\t,\r

s.replace('\n', '').replace('\t','').replace('\r','')
>'  Hello  World Hi '

With regex

s = '''Hello  World \t\n\r\tHi There'''
regex = re.compile(r'[\n\r\t]')
regex.sub("", s)
>'Hello  World Hi There'

with Join

s = '''Hello  World \t\n\r\tHi There'''
' '.join(s.split())
>'Hello  World Hi There'

DeepBlue , 2019-11-06 20:50:30

First split lines then join them by any separator you like.
  x = ' '.join(x.splitlines())

should work like a charm.

user4178860 , 2014-10-24 18:34:12

A catch all:
line = line.rstrip('\r|\n')

Flimm , 2016-06-30 16:20:15

rstrip does not take regular expression. "hi|||\n\n".rstrip("\r|\n") returns "hi"Flimm Jun 30 '16 at 16:20

[Nov 23, 2019] What is a None value?

Jan 01, 2013 | stackoverflow.com

The_Diver ,

I have been studying Python, and I read a chapter which describes the None value, but unfortunately this book isn't very clear at some points. I thought that I would find the answer to my question, if I share it there.

I want to know what the None value is and what do you use it for?

And also, I don't get this part of the book:

Assigning a value of None to a variable is one way to reset it to its original, empty state.

What does that mean?

The answers were great, although I didn't understand most of answers due to my low knowledge of the computer world (I haven't learned about classes, objects, etc.). What does this sentence mean?

Assigning a value of None to a variable is one way to reset it to its original, empty state.

Final:

Finally I've got my answer from looking to different answers. I must appreciate all the people who put their times to help me (especially Martijn Pieters and DSM), and I wish that I could choose all answers as the best, but the selection is limited to one. All of the answers were great.

DSM , 2013-10-20 05:29:23

Martijn's answer explains what None is in Python, and correctly states that the book is misleading. Since Python programmers as a rule would never say

Assigning a value of None to a variable is one way to reset it to its original, empty state.

it's hard to explain what Briggs means in a way which makes sense and explains why no one here seems happy with it. One analogy which may help:

In Python, variable names are like stickers put on objects. Every sticker has a unique name written on it, and it can only be on one object at a time, but you could put more than one sticker on the same object, if you wanted to. When you write

F = "fork"

you put the sticker "F" on a string object "fork" . If you then write

F = None

you move the sticker to the None object.

What Briggs is asking you to imagine is that you didn't write the sticker "F" , there was already an F sticker on the None , and all you did was move it, from None to "fork" . So when you type F = None , you're "reset[ting] it to its original, empty state", if we decided to treat None as meaning empty state .

I can see what he's getting at, but that's a bad way to look at it. If you start Python and type print(F) , you see

>>> print(F)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
NameError: name 'F' is not defined

and that NameError means Python doesn't recognize the name F , because there is no such sticker . If Briggs were right and F = None resets F to its original state, then it should be there now, and we should see

>>> print(F)
None

like we do after we type F = None and put the sticker on None .


So that's all that's going on. In reality, Python comes with some stickers already attached to objects (built-in names), but others you have to write yourself with lines like F = "fork" and A = 2 and c17 = 3.14 , and then you can stick them on other objects later (like F = 10 or F = None ; it's all the same.)

Briggs is pretending that all possible stickers you might want to write were already stuck to the None object.

Martijn Pieters ,

None is just a value that commonly is used to signify 'empty', or 'no value here'. It is a signal object ; it only has meaning because the Python documentation says it has that meaning.

There is only one copy of that object in a given Python interpreter session.

If you write a function, and that function doesn't use an explicit return statement, None is returned instead, for example. That way, programming with functions is much simplified; a function always returns something , even if it is only that one None object.

You can test for it explicitly:

if foo is None:
    # foo is set to None

if bar is not None:
    # bar is set to something *other* than None

Another use is to give optional parameters to functions an 'empty' default:

def spam(foo=None):
    if foo is not None:
        # foo was specified, do something clever!

The function spam() has a optional argument; if you call spam() without specifying it, the default value None is given to it, making it easy to detect if the function was called with an argument or not.

Other languages have similar concepts. SQL has NULL ; JavaScript has undefined and null , etc.

Note that in Python, variables exist by virtue of being used. You don't need to declare a variable first, so there are no really empty variables in Python. Setting a variable to None is then not the same thing as setting it to a default empty value; None is a value too, albeit one that is often used to signal emptyness. The book you are reading is misleading on that point.

thefourtheye , 2013-10-20 03:34:34

This is what the Python documentation has got to say about None :

The sole value of types.NoneType. None is frequently used to represent the absence of a value, as when default arguments are not passed to a function.

Changed in version 2.4: Assignments to None are illegal and raise a SyntaxError.

Note The names None and debug cannot be reassigned (assignments to them, even as an attribute name, raise SyntaxError), so they can be considered "true" constants.

  1. Let's confirm the type of None first
    print type(None)
    print None.__class__
    

    Output

    <type 'NoneType'>
    <type 'NoneType'>
    

Basically, NoneType is a data type just like int , float , etc. You can check out the list of default types available in Python in 8.15. types -- Names for built-in types .

  1. And, None is an instance of NoneType class. So we might want to create instances of None ourselves. Let's try that
    print types.IntType()
    print types.NoneType()
    

    Output

    0
    TypeError: cannot create 'NoneType' instances
    

So clearly, cannot create NoneType instances. We don't have to worry about the uniqueness of the value None .

  1. Let's check how we have implemented None internally.
    print dir(None)
    

    Output

    ['__class__', '__delattr__', '__doc__', '__format__', '__getattribute__', 
     '__hash__', '__init__', '__new__', '__reduce__', '__reduce_ex__',
     '__repr__', '__setattr__', '__sizeof__', '__str__', '__subclasshook__']
    

Except __setattr__ , all others are read-only attributes. So, there is no way we can alter the attributes of None .

  1. Let's try and add new attributes to None
    setattr(types.NoneType, 'somefield', 'somevalue')
    setattr(None, 'somefield', 'somevalue')
    None.somefield = 'somevalue'
    

    Output

    TypeError: can't set attributes of built-in/extension type 'NoneType'
    AttributeError: 'NoneType' object has no attribute 'somefield'
    AttributeError: 'NoneType' object has no attribute 'somefield'
    

The above seen statements produce these error messages, respectively. It means that, we cannot create attributes dynamically on a None instance.

  1. Let us check what happens when we assign something None . As per the documentation, it should throw a SyntaxError . It means, if we assign something to None , the program will not be executed at all.
    None = 1
    

    Output

    SyntaxError: cannot assign to None
    

We have established that

  1. None is an instance of NoneType
  2. None cannot have new attributes
  3. Existing attributes of None cannot be changed.
  4. We cannot create other instances of NoneType
  5. We cannot even change the reference to None by assigning values to it.

So, as mentioned in the documentation, None can really be considered as a true constant .

Happy knowing None :)

kojiro , 2013-10-20 20:15:43

The book you refer to is clearly trying to greatly simplify the meaning of None . Python variables don't have an initial, empty state – Python variables are bound (only) when they're defined . You can't create a Python variable without giving it a value.
>>> print(x)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
NameError: name 'x' is not defined
>>> def test(x):
...   print(x)
... 
>>> test()
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
TypeError: test() takes exactly 1 argument (0 given)
>>> def test():
...   print(x)
... 
>>> test()
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "<stdin>", line 2, in test
NameError: global name 'x' is not defined

but sometimes you want to make a function mean different things depending on whether a variable is defined or not. You can create an argument with a default value of None :

>>> def test(x=None):
...   if x is None:
...     print('no x here')
...   else:
...     print(x)
... 
>>> test()
no x here
>>> test('x!')
x!

The fact that this value is the special None value is not terribly important in this case. I could've used any default value:

>>> def test(x=-1):
...   if x == -1:
...     print('no x here')
...   else:
...     print(x)
... 
>>> test()
no x here
>>> test('x!')
x!

but having None around gives us two benefits:

  1. We don't have to pick a special value like -1 whose meaning is unclear, and
  2. Our function may actually need to handle -1 as a normal input.
>>> test(-1)
no x here

oops!

So the book is a little misleading mostly in its use of the word reset – assigning None to a name is a signal to a programmer that that value isn't being used or that the function should behave in some default way, but to reset a value to its original, undefined state you must use the del keyword:

>>> x = 3
>>> x
3
>>> del x
>>> x
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
NameError: name 'x' is not defined

Jadav Bheda ,

Other answers have already explained meaning of None beautifully. However, I would still like to throw more light on this using an example.

Example:

def extendList(val, list=[]):
    list.append(val)
    return list

list1 = extendList(10)
list2 = extendList(123,[])
list3 = extendList('a')

print "list1 = %s" % list1
print "list2 = %s" % list2
print "list3 = %s" % list3

Now try to guess output of above list. Well, the answer is surprisingly as below:

list1 = [10, 'a']
list2 = [123]
list3 = [10, 'a']

But Why?

Many will mistakenly expect list1 to be equal to [10] and list3 to be equal to ['a'] , thinking that the list argument will be set to its default value of [] each time extendList is called.

However, what actually happens is that the new default list is created only once when the function is defined , and that same list is then used subsequently whenever extendList is invoked without a list argument being specified. This is because expressions in default arguments are calculated when the function is defined, not when it's called .

list1 and list3 are therefore operating on the same default list, whereas list2 is operating on a separate list that it created (by passing its own empty list as the value for the list parameter).


'None' the savior: (Modify example above to produce desired behavior)

def extendList(val, list=None):
    if list is None:
       list = []
    list.append(val)
    return list

list1 = extendList(10)
list2 = extendList(123,[])
list3 = extendList('a')

print "list1 = %s" % list1
print "list2 = %s" % list2
print "list3 = %s" % list3

With this revised implementation, the output would be:

list1 = [10]
list2 = [123]
list3 = ['a']

Note - Example credit to toptal.com

GrvTyagi ,

None is a singleton object (meaning there is only one None), used in many places in the language and library to represent the absence of some other value.


For example:
if d is a dictionary, d.get(k) will return d[k] if it exists, but None if d has no key k .

Read this info from a great blog: http://python-history.blogspot.in/

Azeirah , 2013-10-20 02:15:34

I love code examples (as well as fruit), so let me show you
apple = "apple"
print(apple)
>>> apple
apple = None
print(apple)
>>> None

None means nothing, it has no value.

None evaluates to False.

, 2013-10-20 02:20:22

print(None) prints.. None . It does not , as you suggest, print nothing. – Martijn Pieters ♦ Oct 20 '13 at 2:16

[Nov 22, 2019] Using global variables in a function

Jan 01, 2009 | stackoverflow.com

user46646 , 2009-01-08 05:45:02

How can I create or use a global variable in a function?

If I create a global variable in one function, how can I use that global variable in another function? Do I need to store the global variable in a local variable of the function which needs its access?

Paul Stephenson , 2009-01-08 08:39:44

You can use a global variable in other functions by declaring it as global in each function that assigns to it:
globvar = 0

def set_globvar_to_one():
    global globvar    # Needed to modify global copy of globvar
    globvar = 1

def print_globvar():
    print(globvar)     # No need for global declaration to read value of globvar

set_globvar_to_one()
print_globvar()       # Prints 1

I imagine the reason for it is that, since global variables are so dangerous, Python wants to make sure that you really know that's what you're playing with by explicitly requiring the global keyword.

See other answers if you want to share a global variable across modules.

Jeff Shannon , 2009-01-08 09:19:55

If I'm understanding your situation correctly, what you're seeing is the result of how Python handles local (function) and global (module) namespaces.

Say you've got a module like this:

# sample.py
myGlobal = 5

def func1():
    myGlobal = 42

def func2():
    print myGlobal

func1()
func2()

You might expecting this to print 42, but instead it prints 5. As has already been mentioned, if you add a ' global ' declaration to func1() , then func2() will print 42.

def func1():
    global myGlobal
    myGlobal = 42

What's going on here is that Python assumes that any name that is assigned to , anywhere within a function, is local to that function unless explicitly told otherwise. If it is only reading from a name, and the name doesn't exist locally, it will try to look up the name in any containing scopes (e.g. the module's global scope).

When you assign 42 to the name myGlobal , therefore, Python creates a local variable that shadows the global variable of the same name. That local goes out of scope and is garbage-collected when func1() returns; meanwhile, func2() can never see anything other than the (unmodified) global name. Note that this namespace decision happens at compile time, not at runtime -- if you were to read the value of myGlobal inside func1() before you assign to it, you'd get an UnboundLocalError , because Python has already decided that it must be a local variable but it has not had any value associated with it yet. But by using the ' global ' statement, you tell Python that it should look elsewhere for the name instead of assigning to it locally.

(I believe that this behavior originated largely through an optimization of local namespaces -- without this behavior, Python's VM would need to perform at least three name lookups each time a new name is assigned to inside a function (to ensure that the name didn't already exist at module/builtin level), which would significantly slow down a very common operation.)

gimel , 2009-01-08 05:59:04

You may want to explore the notion of namespaces . In Python, the module is the natural place for global data:

Each module has its own private symbol table, which is used as the global symbol table by all functions defined in the module. Thus, the author of a module can use global variables in the module without worrying about accidental clashes with a user's global variables. On the other hand, if you know what you are doing you can touch a module's global variables with the same notation used to refer to its functions, modname.itemname .

A specific use of global-in-a-module is described here - How do I share global variables across modules? , and for completeness the contents are shared here:

The canonical way to share information across modules within a single program is to create a special configuration module (often called config or cfg ). Just import the configuration module in all modules of your application; the module then becomes available as a global name. Because there is only one instance of each module, any changes made to the module object get reflected everywhere. For example:

File: config.py

x = 0   # Default value of the 'x' configuration setting

File: mod.py

import config
config.x = 1

File: main.py

import config
import mod
print config.x

SingleNegationElimination ,

Python uses a simple heuristic to decide which scope it should load a variable from, between local and global. If a variable name appears on the left hand side of an assignment, but is not declared global, it is assumed to be local. If it does not appear on the left hand side of an assignment, it is assumed to be global.
>>> import dis
>>> def foo():
...     global bar
...     baz = 5
...     print bar
...     print baz
...     print quux
... 
>>> dis.disassemble(foo.func_code)
  3           0 LOAD_CONST               1 (5)
              3 STORE_FAST               0 (baz)

  4           6 LOAD_GLOBAL              0 (bar)
              9 PRINT_ITEM          
             10 PRINT_NEWLINE       

  5          11 LOAD_FAST                0 (baz)
             14 PRINT_ITEM          
             15 PRINT_NEWLINE       

  6          16 LOAD_GLOBAL              1 (quux)
             19 PRINT_ITEM          
             20 PRINT_NEWLINE       
             21 LOAD_CONST               0 (None)
             24 RETURN_VALUE        
>>>

See how baz, which appears on the left side of an assignment in foo() , is the only LOAD_FAST variable.

J S , 2009-01-08 09:03:33

If you want to refer to a global variable in a function, you can use the global keyword to declare which variables are global. You don't have to use it in all cases (as someone here incorrectly claims) - if the name referenced in an expression cannot be found in local scope or scopes in the functions in which this function is defined, it is looked up among global variables.

However, if you assign to a new variable not declared as global in the function, it is implicitly declared as local, and it can overshadow any existing global variable with the same name.

Also, global variables are useful, contrary to some OOP zealots who claim otherwise - especially for smaller scripts, where OOP is overkill.

Rauni Lillemets ,

In addition to already existing answers and to make this more confusing:

In Python, variables that are only referenced inside a function are implicitly global . If a variable is assigned a new value anywhere within the function's body, it's assumed to be a local . If a variable is ever assigned a new value inside the function, the variable is implicitly local, and you need to explicitly declare it as 'global'.

Though a bit surprising at first, a moment's consideration explains this. On one hand, requiring global for assigned variables provides a bar against unintended side-effects. On the other hand, if global was required for all global references, you'd be using global all the time. You'd have to declare as global every reference to a built-in function or to a component of an imported module. This clutter would defeat the usefulness of the global declaration for identifying side-effects.

Source: What are the rules for local and global variables in Python? .

Aaron Hall ,

If I create a global variable in one function, how can I use that variable in another function?

We can create a global with the following function:

def create_global_variable():
    global global_variable # must declare it to be a global first
    # modifications are thus reflected on the module's global scope
    global_variable = 'Foo'

Writing a function does not actually run its code. So we call the create_global_variable function:

>>> create_global_variable()
Using globals without modification

You can just use it, so long as you don't expect to change which object it points to:

For example,

def use_global_variable():
    return global_variable + '!!!'

and now we can use the global variable:

>>> use_global_variable()
'Foo!!!'
Modification of the global variable from inside a function

To point the global variable at a different object, you are required to use the global keyword again:

def change_global_variable():
    global global_variable
    global_variable = 'Bar'

Note that after writing this function, the code actually changing it has still not run:

>>> use_global_variable()
'Foo!!!'

So after calling the function:

>>> change_global_variable()

we can see that the global variable has been changed. The global_variable name now points to 'Bar' :

>>> use_global_variable()
'Bar!!!'

Note that "global" in Python is not truly global - it's only global to the module level. So it is only available to functions written in the modules in which it is global. Functions remember the module in which they are written, so when they are exported into other modules, they still look in the module in which they were created to find global variables.

Local variables with the same name

If you create a local variable with the same name, it will overshadow a global variable:

def use_local_with_same_name_as_global():
    # bad name for a local variable, though.
    global_variable = 'Baz' 
    return global_variable + '!!!'

>>> use_local_with_same_name_as_global()
'Baz!!!'

But using that misnamed local variable does not change the global variable:

>>> use_global_variable()
'Bar!!!'

Note that you should avoid using the local variables with the same names as globals unless you know precisely what you are doing and have a very good reason to do so. I have not yet encountered such a reason.

Bohdan , 2013-10-03 05:41:16

With parallel execution, global variables can cause unexpected results if you don't understand what is happening. Here is an example of using a global variable within multiprocessing. We can clearly see that each process works with its own copy of the variable:
import multiprocessing
import os
import random
import sys
import time

def worker(new_value):
    old_value = get_value()
    set_value(random.randint(1, 99))
    print('pid=[{pid}] '
          'old_value=[{old_value:2}] '
          'new_value=[{new_value:2}] '
          'get_value=[{get_value:2}]'.format(
          pid=str(os.getpid()),
          old_value=old_value,
          new_value=new_value,
          get_value=get_value()))

def get_value():
    global global_variable
    return global_variable

def set_value(new_value):
    global global_variable
    global_variable = new_value

global_variable = -1

print('before set_value(), get_value() = [%s]' % get_value())
set_value(new_value=-2)
print('after  set_value(), get_value() = [%s]' % get_value())

processPool = multiprocessing.Pool(processes=5)
processPool.map(func=worker, iterable=range(15))

Output:

before set_value(), get_value() = [-1]
after  set_value(), get_value() = [-2]
pid=[53970] old_value=[-2] new_value=[ 0] get_value=[23]
pid=[53971] old_value=[-2] new_value=[ 1] get_value=[42]
pid=[53970] old_value=[23] new_value=[ 4] get_value=[50]
pid=[53970] old_value=[50] new_value=[ 6] get_value=[14]
pid=[53971] old_value=[42] new_value=[ 5] get_value=[31]
pid=[53972] old_value=[-2] new_value=[ 2] get_value=[44]
pid=[53973] old_value=[-2] new_value=[ 3] get_value=[94]
pid=[53970] old_value=[14] new_value=[ 7] get_value=[21]
pid=[53971] old_value=[31] new_value=[ 8] get_value=[34]
pid=[53972] old_value=[44] new_value=[ 9] get_value=[59]
pid=[53973] old_value=[94] new_value=[10] get_value=[87]
pid=[53970] old_value=[21] new_value=[11] get_value=[21]
pid=[53971] old_value=[34] new_value=[12] get_value=[82]
pid=[53972] old_value=[59] new_value=[13] get_value=[ 4]
pid=[53973] old_value=[87] new_value=[14] get_value=[70]

user2876408 ,

As it turns out the answer is always simple.

Here is a small sample module with a simple way to show it in a main definition:

def five(enterAnumber,sumation):
    global helper
    helper  = enterAnumber + sumation

def isTheNumber():
    return helper

Here is how to show it in a main definition:

import TestPy

def main():
    atest  = TestPy
    atest.five(5,8)
    print(atest.isTheNumber())

if __name__ == '__main__':
    main()

This simple code works just like that, and it will execute. I hope it helps.

gxyd , 2014-12-04 06:27:43

What you are saying is to use the method like this:
globvar = 5

def f():
    var = globvar
    print(var)

f()  # Prints 5

But the better way is to use the global variable like this:

globavar = 5
def f():
    global globvar
    print(globvar)
f()   #prints 5

Both give the same output.

Mohamed El-Saka , 2014-12-20 12:45:26

You need to reference the global variable in every function you want to use.

As follows:

var = "test"

def printGlobalText():
    global var #wWe are telling to explicitly use the global version
    var = "global from printGlobalText fun."
    print "var from printGlobalText: " + var

def printLocalText():
    #We are NOT telling to explicitly use the global version, so we are creating a local variable
    var = "local version from printLocalText fun"
    print "var from printLocalText: " + var

printGlobalText()
printLocalText()
"""
Output Result:
var from printGlobalText: global from printGlobalText fun.
var from printLocalText: local version from printLocalText
[Finished in 0.1s]
"""

Kylotan , 2009-01-09 11:56:19

You're not actually storing the global in a local variable, just creating a local reference to the same object that your original global reference refers to. Remember that pretty much everything in Python is a name referring to an object, and nothing gets copied in usual operation.

If you didn't have to explicitly specify when an identifier was to refer to a predefined global, then you'd presumably have to explicitly specify when an identifier is a new local variable instead (for example, with something like the 'var' command seen in JavaScript). Since local variables are more common than global variables in any serious and non-trivial system, Python's system makes more sense in most cases.

You could have a language which attempted to guess, using a global variable if it existed or creating a local variable if it didn't. However, that would be very error-prone. For example, importing another module could inadvertently introduce a global variable by that name, changing the behaviour of your program.

Sagar Mehta ,

Try this:
def x1():
    global x
    x = 6

def x2():
    global x
    x = x+1
    print x

x = 5
x1()
x2()  # output --> 7

Martin Thoma , 2017-04-07 18:52:13

In case you have a local variable with the same name, you might want to use the globals() function .
globals()['your_global_var'] = 42

, 2015-10-24 15:46:18

Following on and as an add on, use a file to contain all global variables all declared locally and then import as :

File initval.py :

Stocksin = 300
Prices = []

File getstocks.py :

import initval as iv

def getmystocks(): 
    iv.Stocksin = getstockcount()


def getmycharts():
    for ic in range(iv.Stocksin):

Mike Lampton , 2016-01-07 20:41:19

Writing to explicit elements of a global array does not apparently need the global declaration, though writing to it "wholesale" does have that requirement:
import numpy as np

hostValue = 3.14159
hostArray = np.array([2., 3.])
hostMatrix = np.array([[1.0, 0.0],[ 0.0, 1.0]])

def func1():
    global hostValue    # mandatory, else local.
    hostValue = 2.0

def func2():
    global hostValue    # mandatory, else UnboundLocalError.
    hostValue += 1.0

def func3():
    global hostArray    # mandatory, else local.
    hostArray = np.array([14., 15.])

def func4():            # no need for globals
    hostArray[0] = 123.4

def func5():            # no need for globals
    hostArray[1] += 1.0

def func6():            # no need for globals
    hostMatrix[1][1] = 12.

def func7():            # no need for globals
    hostMatrix[0][0] += 0.33

func1()
print "After func1(), hostValue = ", hostValue
func2()
print "After func2(), hostValue = ", hostValue
func3()
print "After func3(), hostArray = ", hostArray
func4()
print "After func4(), hostArray = ", hostArray
func5()
print "After func5(), hostArray = ", hostArray
func6()
print "After func6(), hostMatrix = \n", hostMatrix
func7()
print "After func7(), hostMatrix = \n", hostMatrix

Rafaël Dera ,

I'm adding this as I haven't seen it in any of the other answers and it might be useful for someone struggling with something similar. The globals() function returns a mutable global symbol dictionary where you can "magically" make data available for the rest of your code. For example:
from pickle import load
def loaditem(name):
    with open(r"C:\pickle\file\location"+"\{}.dat".format(name), "rb") as openfile:
        globals()[name] = load(openfile)
    return True

and

from pickle import dump
def dumpfile(name):
    with open(name+".dat", "wb") as outfile:
        dump(globals()[name], outfile)
    return True

Will just let you dump/load variables out of and into the global namespace. Super convenient, no muss, no fuss. Pretty sure it's Python 3 only.

llewellyn falco , 2017-08-19 08:48:27

Reference the class namespace where you want the change to show up.

In this example, runner is using max from the file config. I want my test to change the value of max when runner is using it.

main/config.py

max = 15000

main/runner.py

from main import config
def check_threads():
    return max < thread_count

tests/runner_test.py

from main import runner                # <----- 1. add file
from main.runner import check_threads
class RunnerTest(unittest):
   def test_threads(self):
       runner.max = 0                  # <----- 2. set global 
       check_threads()

[Nov 22, 2019] python - Using global variables in a function - Stack Overflow

Nov 22, 2019 | stackoverflow.com

Paul Uszak ,Sep 22 at 22:57

If you want to refer to a global variable in a function, you can use the global keyword to declare which variables are global. You don't have to use it in all cases (as someone here incorrectly claims) - if the name referenced in an expression cannot be found in local scope or scopes in the functions in which this function is defined, it is looked up among global variables.

However, if you assign to a new variable not declared as global in the function, it is implicitly declared as local, and it can overshadow any existing global variable with the same name.

Also, global variables are useful, contrary to some OOP zealots who claim otherwise - especially for smaller scripts, where OOP is overkill.

[Nov 21, 2019] How to fix Python indentation

Nov 21, 2019 | stackoverflow.com

ephemient ,Feb 28, 2010 at 0:35

Use the reindent.py script that you find in the Tools/scripts/ directory of your Python installation:

Change Python (.py) files to use 4-space indents and no hard tab characters. Also trim excess spaces and tabs from ends of lines, and remove empty lines at the end of files. Also ensure the last line ends with a newline.

Have a look at that script for detailed usage instructions.

[Nov 21, 2019] Python Formatter Tool

Jul 3, 2014 | stackoverflow.com

tricasse ,Jul 3, 2014 at 19:15

I was wondering if there exists a sort of Python beautifier like the gnu-indent command line tool for C code. Of course indentation is not the point in Python since it is programmer's responsibility but I wish to get my code written in a perfectly homogenous way, taking care particularly of having always identical blank space between operands or after and before separators and between blocks.

Mike A ,Mar 1, 2010 at 17:49

I am the one who asks the question. In fact, the tool the closest to my needs seems to be PythonTidy (it's a Python program of course : Python is best served by himself ;) ).

tom ,Sep 29, 2014 at 18:26

autopep8 attempts to automate making your code conform to pep8 coding standards

https://pypi.python.org/pypi/autopep8

Eyal Levin ,Oct 31, 2017 at 9:31

You can also try yapf :

A formatter for Python files

https://github.com/google/yapf/

Vinko Vrsalovic ,Jun 23, 2009 at 12:57

PyLint has some formatting checks.

xxx ,Jun 23, 2009 at 12:57

Have you looked at pindent ?

[Nov 21, 2019] Passing an Array/List into Python

Dec 17, 2017 | stackoverflow.com

JAL ,Dec 17, 2017 at 8:12

I've been looking at passing arrays, or lists, as Python tends to call them, into a function.

I read something about using *args, such as:

def someFunc(*args)
    for x in args
        print x

But not sure if this is right/wrong. Nothing seems to work as I want. I'm used to be able to pass arrays into PHP function with ease and this is confusing me. It also seems I can't do this:

def someFunc(*args, someString)

As it throws up an error.

I think I've just got myself completely confused and looking for someone to clear it up for me.

Rafał Rawicki ,Feb 13 at 15:08

When you define your function using this syntax:
def someFunc(*args)
    for x in args
        print x

You're telling it that you expect a variable number of arguments. If you want to pass in a List (Array from other languages) you'd do something like this:

def someFunc(myList = [], *args)
    for x in myList:
        print x

Then you can call it with this:

items = [1,2,3,4,5]

someFunc(items)

You need to define named arguments before variable arguments, and variable arguments before keyword arguments. You can also have this:

def someFunc(arg1, arg2, arg3, *args, **kwargs)
    for x in args
        print x

Which requires at least three arguments, and supports variable numbers of other arguments and keyword arguments.

JoshD ,Oct 18, 2010 at 20:28

You can pass lists just like other types:
l = [1,2,3]

def stuff(a):
   for x in a:
      print a


stuff(l)

This prints the list l. Keep in mind lists are passed as references not as a deep copy.

Gintautas Miliauskas ,Oct 18, 2010 at 16:14

Python lists (which are not just arrays because their size can be changed on the fly) are normal Python objects and can be passed in to functions as any variable. The * syntax is used for unpacking lists, which is probably not something you want to do now.

,

You don't need to use the asterisk to accept a list.

Simply give the argument a name in the definition, and pass in a list like

def takes_list(a_list):
    for item in a_list:
         print item

[Nov 21, 2019] How to print to stderr in Python?

Nov 21, 2019 | stackoverflow.com

Ask Question Asked 8 years, 7 months ago Active 3 months ago Viewed 762k times 1239 170


Steve Howard ,Jul 31, 2013 at 14:05

There are several ways to write to stderr:
# Note: this first one does not work in Python 3
print >> sys.stderr, "spam"

sys.stderr.write("spam\n")

os.write(2, b"spam\n")

from __future__ import print_function
print("spam", file=sys.stderr)

That seems to contradict zen of Python #13 , so what's the difference here and are there any advantages or disadvantages to one way or the other? Which way should be used?

There should be one -- and preferably only one -- obvious way to do it.

Dan H ,May 16, 2017 at 22:51

I found this to be the only one short + flexible + portable + readable:
from __future__ import print_function
import sys

def eprint(*args, **kwargs):
    print(*args, file=sys.stderr, **kwargs)

The function eprint can be used in the same way as the standard print function:

>>> print("Test")
Test
>>> eprint("Test")
Test
>>> eprint("foo", "bar", "baz", sep="---")
foo---bar---baz

Dheeraj V.S. ,Jan 13, 2013 at 3:18

import sys
sys.stderr.write()

Is my choice, just more readable and saying exactly what you intend to do and portable across versions.

Edit: being 'pythonic' is a third thought to me over readability and performance... with these two things in mind, with python 80% of your code will be pythonic. list comprehension being the 'big thing' that isn't used as often (readability).

Michael Scheper ,Aug 26 at 17:01

print >> sys.stderr is gone in Python3. http://docs.python.org/3.0/whatsnew/3.0.html says:
Old: print >>sys.stderr, "fatal error"
New: print("fatal error", file=sys.stderr)

For many of us, it feels somewhat unnatural to relegate the destination to the end of the command. The alternative

sys.stderr.write("fatal error\n")

looks more object oriented, and elegantly goes from the generic to the specific. But note that write is not a 1:1 replacement for print .

luketparkinson ,Apr 23, 2013 at 10:04

For Python 2 my choice is: print >> sys.stderr, 'spam' Because you can simply print lists/dicts etc. without convert it to string. print >> sys.stderr, {'spam': 'spam'} instead of: sys.stderr.write(str({'spam': 'spam'}))

Mnebuerquo ,Jul 11 at 9:44

Nobody's mentioned logging yet, but logging was created specifically to communicate error messages. By default it is set up to write to stderr. This script:
# foo.py
import logging
logging.basicConfig(format='%(message)s')

logging.warning('I print to stderr by default')
logging.info('For this you must change the level and add a handler.')
print('hello world')

has the following result when run on the command line:

$ python3 foo.py > bar.txt
I print to stderr by default

(and bar.txt contains the 'hello world')

(Note, logging.warn has been deprecated , use logging.warning instead)

porgarmingduod ,Apr 15, 2016 at 1:37

I would say that your first approach:
print >> sys.stderr, 'spam'

is the "One . . . obvious way to do it" The others don't satisfy rule #1 ("Beautiful is better than ugly.")

Rebs ,Dec 30, 2013 at 2:26

I did the following using Python 3:
from sys import stderr

def print_err(*args, **kwargs):
    print(*args, file=stderr, **kwargs)

So now I'm able to add keyword arguments, for example, to avoid carriage return:

print_err("Error: end of the file reached. The word ", end='')
print_err(word, "was not found")

AMS ,Nov 5, 2015 at 14:15

This will mimic the standard print function but output on stderr
def print_err(*args):
    sys.stderr.write(' '.join(map(str,args)) + '\n')

Agi Hammerthief ,Dec 31, 2015 at 22:58

EDIT In hind-sight, I think the potential confusion with changing sys.stderr and not seeing the behaviour updated makes this answer not as good as just using a simple function as others have pointed out.

Using partial only saves you 1 line of code. The potential confusion is not worth saving 1 line of code.

original

To make it even easier, here's a version that uses 'partial', which is a big help in wrapping functions.

from __future__ import print_function
import sys
from functools import partial

error = partial(print, file=sys.stderr)

You then use it like so

error('An error occured!')

You can check that it's printing to stderr and not stdout by doing the following (over-riding code from http://coreygoldberg.blogspot.com.au/2009/05/python-redirect-or-turn-off-stdout-and.html ):

# over-ride stderr to prove that this function works.
class NullDevice():
    def write(self, s):
        pass
sys.stderr = NullDevice()

# we must import print error AFTER we've removed the null device because
# it has been assigned and will not be re-evaluated.
# assume error function is in print_error.py
from print_error import error

# no message should be printed
error("You won't see this error!")

The downside to this is partial assigns the value of sys.stderr to the wrapped function at the time of creation. Which means, if you redirect stderr later it won't affect this function. If you plan to redirect stderr, then use the **kwargs method mentioned by aaguirre on this page.

Florian Castellane ,Jan 8 at 6:57

In Python 3, one can just use print():
print(*objects, sep=' ', end='\n', file=sys.stdout, flush=False)

almost out of the box:

import sys
print("Hello, world!", file=sys.stderr)

or:

from sys import stderr
print("Hello, world!", file=stderr)

This is straightforward and does not need to include anything besides sys.stderr .

phoenix ,Mar 2, 2016 at 23:57

The same applies to stdout:
print 'spam'
sys.stdout.write('spam\n')

As stated in the other answers, print offers a pretty interface that is often more convenient (e.g. for printing debug information), while write is faster and can also be more convenient when you have to format the output exactly in certain way. I would consider maintainability as well:

  1. You may later decide to switch between stdout/stderr and a regular file.
  2. print() syntax has changed in Python 3, so if you need to support both versions, write() might be better.

user1928764 ,Feb 10, 2016 at 2:29

I am working in python 3.4.3. I am cutting out a little typing that shows how I got here:
[18:19 jsilverman@JSILVERMAN-LT7 pexpect]$ python3
>>> import sys
>>> print("testing", file=sys.stderr)
testing
>>>
[18:19 jsilverman@JSILVERMAN-LT7 pexpect]$

Did it work? Try redirecting stderr to a file and see what happens:

[18:22 jsilverman@JSILVERMAN-LT7 pexpect]$ python3 2> /tmp/test.txt
>>> import sys
>>> print("testing", file=sys.stderr)
>>> [18:22 jsilverman@JSILVERMAN-LT7 pexpect]$
[18:22 jsilverman@JSILVERMAN-LT7 pexpect]$ cat /tmp/test.txt
Python 3.4.3 (default, May  5 2015, 17:58:45)
[GCC 4.9.2] on cygwin
Type "help", "copyright", "credits" or "license" for more information.
testing

[18:22 jsilverman@JSILVERMAN-LT7 pexpect]$

Well, aside from the fact that the little introduction that python gives you has been slurped into stderr (where else would it go?), it works.

hamish ,Oct 8, 2017 at 16:18

If you do a simple test:
import time
import sys

def run1(runs):
    x = 0
    cur = time.time()
    while x < runs:
        x += 1
        print >> sys.stderr, 'X'
    elapsed = (time.time()-cur)
    return elapsed

def run2(runs):
    x = 0
    cur = time.time()
    while x < runs:
        x += 1
        sys.stderr.write('X\n')
        sys.stderr.flush()
    elapsed = (time.time()-cur)
    return elapsed

def compare(runs):
    sum1, sum2 = 0, 0
    x = 0
    while x < runs:
        x += 1
        sum1 += run1(runs)
        sum2 += run2(runs)
    return sum1, sum2

if __name__ == '__main__':
    s1, s2 = compare(1000)
    print "Using (print >> sys.stderr, 'X'): %s" %(s1)
    print "Using (sys.stderr.write('X'),sys.stderr.flush()):%s" %(s2)
    print "Ratio: %f" %(float(s1) / float(s2))

You will find that sys.stderr.write() is consistently 1.81 times faster!

Vinay Kumar ,Jan 30, 2018 at 13:17

Answer to the question is : There are different way to print stderr in python but that depends on 1.) which python version we are using 2.) what exact output we want.

The differnce between print and stderr's write function: stderr : stderr (standard error) is pipe that is built into every UNIX/Linux system, when your program crashes and prints out debugging information (like a traceback in Python), it goes to the stderr pipe.

print : print is a wrapper that formats the inputs (the input is the space between argument and the newline at the end) and it then calls the write function of a given object, the given object by default is sys.stdout, but we can pass a file i.e we can print the input in a file also.

Python2: If we are using python2 then

>>> import sys
>>> print "hi"
hi
>>> print("hi")
hi
>>> print >> sys.stderr.write("hi")
hi

Python2 trailing comma has in Python3 become a parameter, so if we use trailing commas to avoid the newline after a print, this will in Python3 look like print('Text to print', end=' ') which is a syntax error under Python2.

http://python3porting.com/noconv.html

If we check same above sceario in python3:

>>> import sys
>>> print("hi")
hi

Under Python 2.6 there is a future import to make print into a function. So to avoid any syntax errors and other differences we should start any file where we use print() with from future import print_function. The future import only works under Python 2.6 and later, so for Python 2.5 and earlier you have two options. You can either convert the more complex print to something simpler, or you can use a separate print function that works under both Python2 and Python3.

>>> from __future__ import print_function
>>> 
>>> def printex(*args, **kwargs):
...     print(*args, file=sys.stderr, **kwargs)
... 
>>> printex("hii")
hii
>>>

Case: Point to be noted that sys.stderr.write() or sys.stdout.write() ( stdout (standard output) is a pipe that is built into every UNIX/Linux system) is not a replacement for print, but yes we can use it as a alternative in some case. Print is a wrapper which wraps the input with space and newline at the end and uses the write function to write. This is the reason sys.stderr.write() is faster.

Note: we can also trace and debugg using Logging

#test.py
import logging
logging.info('This is the existing protocol.')
FORMAT = "%(asctime)-15s %(clientip)s %(user)-8s %(message)s"
logging.basicConfig(format=FORMAT)
d = {'clientip': '192.168.0.1', 'user': 'fbloggs'}
logging.warning("Protocol problem: %s", "connection reset", extra=d)

https://docs.python.org/2/library/logging.html#logger-objects

[Nov 21, 2019] Python 3.3 Access string internal representation - Stack Overflow

Nov 21, 2019 | stackoverflow.com

> ,

The internal switch to a more space-efficient storage for unicode values introduced by PEP-393 were made for performance reasons only.

As such they have zero impact on how encoding from and decoding to unicode str values works in Python code. There is absolutely no point in accessing the internal representation from Python. The character A is either stored as 41 , 4100 or 41000000 , depending on how much space the highest codepoint in the string demands, but it'll still be encoded to 41 in ASCII, Latin-1 or UTF-8.

Unless you are writing a C extension that has to deal with this internal representation, there is absolutely no need to worry about how Python actually stored the data.

To debug encoding or decoding issues, I'd use the ascii() function to represent a string using only ASCII codepoints and Python string literal escapes, or you can use the ord() function for turning individual characters into an integer for each codepoint.

For bytes values, the binascii.hexlify() function also comes in handy to quickly turn a series of bytes into their hex representations.

[Nov 21, 2019] What is internal representation of string in Python 3.x - Stack Overflow

Nov 21, 2019 | stackoverflow.com

What is internal representation of string in Python 3.x Ask Question Asked 9 years, 11 months ago Active 8 months ago Viewed 13k times 24 3


thebat ,Dec 3, 2009 at 6:57

In Python 3.x, a string consists of items of Unicode ordinal. (See the quotation from the language reference below.) What is the internal representation of Unicode string? Is it UTF-16?

The items of a string object are Unicode code units. A Unicode code unit is represented by a string object of one item and can hold either a 16-bit or 32-bit value representing a Unicode ordinal (the maximum value for the ordinal is given in sys.maxunicode, and depends on how Python is configured at compile time). Surrogate pairs may be present in the Unicode object, and will be reported as two separate items.

Joachim Sauer ,May 6, 2015 at 10:08

There has been NO CHANGE in Unicode internal representation between Python 2.X and 3.X.

It's definitely NOT UTF-16. UTF-anything is a byte-oriented EXTERNAL representation.

Each code unit (character, surrogate, etc) has been assigned a number from range(0, 2 ** 21). This is called its "ordinal".

Really, the documentation you quoted says it all. Most Python binaries use 16-bit ordinals which restricts you to the Basic Multilingual Plane ("BMP") unless you want to muck about with surrogates (handy if you can't find your hair shirt and your bed of nails is off being de-rusted). For working with the full Unicode repertoire, you'd prefer a "wide build" (32 bits wide).

Briefly, the internal representation in a unicode object is an array of 16-bit unsigned integers, or an array of 32-bit unsigned integers (using only 21 bits).

gwideman ,Oct 24, 2017 at 3:42

The internal representation will change in Python 3.3 which implements PEP 393 . The new representation will pick one or several of ascii, latin-1, utf-8, utf-16, utf-32, generally trying to get a compact representation.

Implicit conversions into surrogate pairs will only be done when talking to legacy APIs (those only exist on windows, where wchar_t is two bytes); the Python string will be preserved. Here are the release notes .

Matthew Brett ,Feb 28, 2015 at 20:32

In Python 3.3 and above, the internal representation of the string will depend on the string, and can be any of latin-1, UCS-2 or UCS-4, as described in PEP 393 .

For previous Pythons, the internal representation depends on the build flags of Python. Python can be built with flag values --enable-unicode=ucs2 or --enable-unicode=ucs4 . ucs2 builds do in fact use UTF-16 as their internal representation , and ucs4 builds use UCS-4 / UTF-32.

codeape ,Dec 3, 2009 at 9:25

Looking at the source code for CPython 3.1.5, in Include/unicodeobject.h :
/* --- Unicode Type ------------------------------------------------------- */

typedef struct {
    PyObject_HEAD
    Py_ssize_t length;          /* Length of raw Unicode data in buffer */
    Py_UNICODE *str;            /* Raw Unicode buffer */
    long hash;                  /* Hash value; -1 if not set */
    int state;                  /* != 0 if interned. In this case the two
                                 * references from the dictionary to this object
                                 * are *not* counted in ob_refcnt. */
    PyObject *defenc;           /* (Default) Encoded version as Python
                                   string, or NULL; this is used for
                                   implementing the buffer protocol */
} PyUnicodeObject;

The characters are stored as an array of Py_UNICODE . On most platforms, I believe Py_UNICODE is #define d as wchar_t .

Ned Deily ,Dec 3, 2009 at 7:12

It depends: see here . This is still true for Python 3 as far as internal representation goes.

YOU ,Dec 3, 2009 at 7:18

I think, Its hard to judge difference between UTF-16, which is just a sequences of 16 bit words, to Python's string object.

And If python is compiled with Unicode=UCS4 option, it will be comparing between UTF-32 and Python string.

So, better consider, they are in different category, although you can transform each others.

,

>>> import array; s = 'Привет мир!'; b = array.array('u', s).tobytes(); print(b); print(len(s) * 4 == len(b))
b'\x1f\x04\x00\x00@\x04\x00\x008\x04\x00\x002\x04\x00\x005\x04\x00\x00B\x04\x00\x00 \x00\x00\x00<\x04\x00\x008\x04\x00\x00@\x04\x00\x00!\x00\x00\x00'
True
>>> import array; s = 'test'; b = array.array('u', s).tobytes(); print(b); print(len(s) * 4 == len(b))
b't\x00\x00\x00e\x00\x00\x00s\x00\x00\x00t\x00\x00\x00'
True
>>>

[Nov 21, 2019] How do I convert a binary string to a number in Perl - Stack Overflow

Nov 21, 2019 | stackoverflow.com

How do I convert a binary string to a number in Perl? Ask Question Asked 10 years, 10 months ago Active 1 year, 9 months ago Viewed 32k times 29 15


Nathan Fellman ,Jan 27, 2009 at 14:43

How can I convert the binary string $x_bin="0001001100101" to its numeric value $x_num=613 in Perl?

innaM ,Jan 28, 2009 at 0:04

sub bin2dec {
    return unpack("N", pack("B32", substr("0" x 32 . shift, -32)));
}

ysth ,Jan 28, 2009 at 1:48

My preferred way is:
$x_num = oct("0b" . $x_bin);

Quoting from man perlfunc :

    oct EXPR
    oct     Interprets EXPR as an octal string and returns the
            corresponding value. (If EXPR happens to start
            off with "0x", interprets it as a hex string. If
            EXPR starts off with "0b", it is interpreted as a
            binary string. Leading whitespace is ignored in
            all three cases.)

innaM ,Jan 27, 2009 at 16:23

As usual, there's is also an excellent CPAN module that should be mentioned here: Bit::Vector .

The transformation would look something like this:

use Bit::Vector;

my $v = Bit::Vector->new_Bin( 32, '0001001100101' );
print "hex: ", $v->to_Hex(), "\n";
print "dec: ", $v->to_Dec(), "\n";

The binary strings can be of almost any length and you can do other neat stuff like bit-shifting, etc.

noswonky ,Jan 28, 2009 at 0:28

Actually you can just stick '0b' on the front and it's treated as a binary number.
perl -le 'print 0b101'
5

But this only works for a bareword.

,

You can use the eval() method to work around the bare-word restriction:
eval "\$num=0b$str;";

[Nov 21, 2019] regex - Perl regular expression match nested brackets - Stack Overflow

Nov 21, 2019 | stackoverflow.com

Perl regular expression: match nested brackets Ask Question Asked 6 years, 8 months ago Active 6 years ago Viewed 21k times 9 7


i-- ,Mar 8, 2013 at 19:33

I'm trying to match nested {} brackets with a regular expressions in Perl so that I can extract certain pieces of text from a file. This is what I have currently:
my @matches = $str =~ /\{(?:\{.*\}|[^\{])*\}|\w+/sg;

foreach (@matches) {
    print "$_\n";
}

At certain times this works as expected. For instance, if $str = "abc {{xyz} abc} {xyz}" I obtain:

abc
{{xyz} abc}
{xyz}

as expected. But for other input strings it does not function as expected. For example, if $str = "{abc} {{xyz}} abc" , the output is:

{abc} {{xyz}}
abc

which is not what I expected. I would have wanted {abc} and {{xyz}} to be on separate lines, since each is balanced on its own in terms of brackets. Is there an issue with my regular expression? If so, how would I go about fixing it?

l--''''''---------'''''''''''' ,Dec 9, 2013 at 14:30

You were surprised how your pattern matched, but noone explained it? Here's how your pattern is matching:
my @matches = $str =~ /\{(?:\{.*\}|[^{])*\}|\w+/sg;
                       ^    ^ ^ ^  ^      ^
                       |    | | |  |      |
{ ---------------------+    | | |  |      |
a --------------------------)-)-)--+      |
b --------------------------)-)-)--+      |
c --------------------------)-)-)--+      |
} --------------------------)-)-)--+      |
  --------------------------)-)-)--+      |
{ --------------------------+ | |         |
{ ----------------------------+ |         |
x ----------------------------+ |         |
y ----------------------------+ |         |
z ----------------------------+ |         |
} ------------------------------+         |
} ----------------------------------------+

As you can see, the problem is that / \{.*\} / matches too much. What should be in there is a something that matches

(?: \s* (?: \{ ... \} | \w+ ) )*

where the ... is

(?: \s* (?: \{ ... \} | \w+ ) )*

So you need some recursion. Named groups are an easy way of doing this.

say $1
   while /
      \G \s*+ ( (?&WORD) | (?&BRACKETED) )

      (?(DEFINE)
         (?<WORD>      \s* \w+ )
         (?<BRACKETED> \s* \{ (?&TEXT)? \s* \} )
         (?<TEXT>      (?: (?&WORD) | (?&BRACKETED) )+ )
      )
   /xg;

But instead of reinventing the wheel, why not use Text::Balanced .

Schwern ,Mar 8, 2013 at 20:01

The problem of matching balanced and nested delimiters is covered in perlfaq5 and I'll leave it to them to cover all the options including (?PARNO) and Regexp::Common .

But matching balanced items is tricky and prone to error, unless you really want to learn and maintain advanced regexes, leave it to a module. Fortunately there is Text::Balanced to handle this and so very much more. It is the Swiss Army Chainsaw of balanced text matching.

Unfortunately it does not handle escaping on bracketed delimiters .

use v5.10;
use strict;
use warnings;

use Text::Balanced qw(extract_multiple extract_bracketed);

my @strings = ("abc {{xyz} abc} {xyz}", "{abc} {{xyz}} abc");

for my $string (@strings) {
    say "Extracting from $string";

    # Extract all the fields, rather than one at a time.
    my @fields = extract_multiple(
        $string,
        [
            # Extract {...}
            sub { extract_bracketed($_[0], '{}') },
            # Also extract any other non whitespace
            qr/\S+/
        ],
        # Return all the fields
        undef,
        # Throw out anything which does not match
        1
    );

    say join "\n", @fields;
    print "\n";
}

You can think of extract_multiple like a more generic and powerful split .

arshajii ,Mar 8, 2013 at 20:34

You need a recursive regex. This should work:
my @matches;
push @matches, $1 while $str =~ /( [^{}\s]+ | ( \{ (?: [^{}]+ | (?2) )* \} ) )/xg;

or, if you prefer the non-loop version:

my @matches = $str =~ /[^{}\s]+ | \{ (?: (?R) | [^{}]+ )+ \} /gx;

nhahtdh ,Mar 8, 2013 at 20:40

To match nested brackets with just one pair at each level of nesting,
but any number of levels, e.g. {1{2{3}}} , you could use
/\{[^}]*[^{]*\}|\w+/g

To match when there may be multiple pairs at any level of nesting, e.g. {1{2}{2}{2}} , you could use

/(?>\{(?:[^{}]*|(?R))*\})|\w+/g

The (?R) is used to match the whole pattern recursively.

To match the text contained within a pair of brackets the engine must match (?:[^{}]*|(?R))* ,
i.e. either [^{}]* or (?R) , zero or more times * .

So in e.g. "{abc {def}}" , after the opening "{" is matched, the [^{}]* will match the "abc " and the (?R) will match the "{def}" , then the closing "}" will be matched.

The "{def}" is matched because (?R) is simply short for the whole pattern
(?>\{(?:[^{}]*|(?R))*\})|\w+ , which as we have just seen will match a "{" followed by text matching [^{}]* , followed by "}" .

Atomic grouping (?> ... ) is used to prevent the regex engine backtracking into bracketed text once it has been matched. This is important to ensure the regex will fail fast if it cannot find a match.

Francisco Zarabozo ,Mar 15, 2013 at 13:15

Wow. What a bunch of complicated answers to something that simple.

The problem you're having is that you're matching in greedy mode. That is, you are aking the regex engine to match as much as possible while making the expression true.

To avoid greedy match, just add a '?' after your quantifier. That makes the match as short as possible.

So, I changed your expression from:

my @matches = $str =~ /\{(?:\{.*\}|[^\{])*\}|\w+/sg;

To:

my @matches = $str =~ /\{(?:\{.*?\}|[^\{])*?\}|\w+/sg;

...and now it works exactly as you're expecting.

HTH

Francisco

Joel Berger ,Mar 8, 2013 at 20:01

One way using the built-in module Text::Balanced .

Content of script.pl :

#!/usr/bin/env perl

use warnings;
use strict;
use Text::Balanced qw<extract_bracketed>;

while ( <DATA> ) { 

    ## Remove '\n' from input string.
    chomp;

    printf qq|%s\n|, $_; 
    print "=" x 20, "\n";


    ## Extract all characters just before first curly bracket.
    my @str_parts = extract_bracketed( $_, '{}', '[^{}]*' );

    if ( $str_parts[2] ) { 
        printf qq|%s\n|, $str_parts[2];
    }   

    my $str_without_prefix = "@str_parts[0,1]";


    ## Extract data of balanced curly brackets, remove leading and trailing
    ## spaces and print.
    while ( my $match = extract_bracketed( $str_without_prefix, '{}' ) ) { 
        $match =~ s/^\s+//;
        $match =~ s/\s+$//;
        printf qq|%s\n|, $match;

    }   

    print "\n";
}

__DATA__
abc {{xyz} abc} {xyz}
{abc} {{xyz}} abc

Run it like:

perl script.pl

That yields:

abc {{xyz} abc} {xyz}
====================
abc 
{{xyz} abc}
{xyz}

{abc} {{xyz}} abc
====================
{abc}
{{xyz}}

Borodin ,Mar 8, 2013 at 19:54

Just modifies and extends the classic solution a bit:
(\{(?:(?1)|[^{}]*+)++\})|[^{}\s]++

Demo (This is in PCRE. The behavior is slightly different from Perl when it comes to recursive regex, but I think it should produce the same result for this case).

After some struggle (I am not familiar with Perl!), this is the demo on ideone . $& refers to the string matched by the whole regex.

my $str = "abc {{xyz} abc} {xyz} {abc} {{xyz}} abc";

while ($str =~ /(\{(?:(?1)|[^{}]*+)++\})|[^{}\s]++/g) {
    print "$&\n"
}

Note that this solution assumes that the input is valid. It will behave rather randomly on invalid input. It can be modified slightly to halt when invalid input is encountered. For that, I need more details on the input format (preferably as a grammar), such as whether abc{xyz}asd is considered valid input or not.

[Nov 21, 2019] regex - How can I extract a string between matching braces in Perl - Stack Overflow

Nov 21, 2019 | stackoverflow.com

How can I extract a string between matching braces in Perl? Ask Question Asked 9 years, 7 months ago Active 2 years, 6 months ago Viewed 18k times 10 3


Srilesh ,Apr 23, 2010 at 17:26

My input file is as below :
HEADER 
{ABC|*|DEF {GHI 0 1 0} {{Points {}}}}

{ABC|*|DEF {GHI 0 2 0} {{Points {}}}}

{ABC|*|XYZ:abc:def {GHI 0 22 0} {{Points {{F1 1.1} {F2 1.2} {F3 1.3} {F4 1.4}}}}}

{ABC|*|XYZ:ghi:jkl {JKL 0 372 0} {{Points {}}}}

{ABC|*|XYZ:mno:pqr {GHI 0 34 0} {{Points {}}}}

{
    ABC|*|XYZ:abc:pqr {GHI 0 68 0}
        {{Points {{F1 11.11} {F2 12.10} {F3 14.11} {F4 16.23}}}}
        }
TRAILER

I want to extract the file into an array as below :

$array[0] = "{ABC|*|DEF {GHI 0 1 0} {{Points {}}}}"

$array[1] = "{ABC|*|DEF {GHI 0 2 0} {{Points {}}}}"

$array[2] = "{ABC|*|XYZ:abc:def {GHI 0 22 0} {{Points {{F1 1.1} {F2 1.2} {F3 1.3} {F4 1.4}}}}}"

..
..

$array[5] = "{
    ABC|*|XYZ:abc:pqr {GHI 0 68 0}
        {{Points {{F1 11.11} {F2 12.10} {F3 14.11} {F4 16.23}}}}
        }"

Which means, I need to match the first opening curly brace with its closing curly brace and extract the string in between.

I have checked the below link, but this doesnt apply to my question. Regex to get string between curly braces "{I want what's between the curly braces}"

I am trying but would really help if someone can assist me with their expertise ...

Thanks Sri ...

Srilesh ,Apr 23, 2010 at 20:26

This can certainly be done with regex at least in modern versions of Perl:
my @array = $str =~ /( \{ (?: [^{}]* | (?0) )* \} )/xg;

print join "\n" => @array;

The regex matches a curly brace block that contains either non curly brace characters, or a recursion into itself (matches nested braces)

Edit: the above code works in Perl 5.10+, for earlier versions the recursion is a bit more verbose:

my $re; $re = qr/ \{ (?: [^{}]* | (??{$re}) )* \} /x;

my @array = $str =~ /$re/xg;

Srilesh ,Apr 23, 2010 at 18:34

Use Text::Balanced

Srilesh ,Apr 23, 2010 at 18:29

I second ysth's suggestion to use the Text::Balanced module. A few lines will get you on your way.
use strict;
use warnings;
use Text::Balanced qw/extract_multiple extract_bracketed/;

my $file;
open my $fileHandle, '<', 'file.txt';

{ 
  local $/ = undef; # or use File::Slurp
  $file = <$fileHandle>;
}

close $fileHandle;

my @array = extract_multiple(
                               $file,
                               [ sub{extract_bracketed($_[0], '{}')},],
                               undef,
                               1
                            );

print $_,"\n" foreach @array;

OUTPUT
{ABC|*|DEF {GHI 0 1 0} {{Points {}}}}
{ABC|*|DEF {GHI 0 2 0} {{Points {}}}}
{ABC|*|XYZ:abc:def {GHI 0 22 0} {{Points {{F1 1.1} {F2 1.2} {F3 1.3} {F4 1.4}}}}}
{ABC|*|XYZ:ghi:jkl {JKL 0 372 0} {{Points {}}}}
{ABC|*|XYZ:mno:pqr {GHI 0 34 0} {{Points {}}}}
{
    ABC|*|XYZ:abc:pqr {GHI 0 68 0}
        {{Points {{F1 11.11} {F2 12.10} {F3 14.11} {F4 16.23}}}}
        }

Srilesh ,Apr 23, 2010 at 18:19

You can always count braces:
my $depth = 0;
my $out = "";
my @list=();
foreach my $fr (split(/([{}])/,$data)) {
    $out .= $fr;
    if($fr eq '{') {
        $depth ++;
    }
    elsif($fr eq '}') {
        $depth --;
        if($depth ==0) {
            $out =~ s/^.*?({.*}).*$/$1/s; # trim
            push @list, $out;
            $out = "";
        }
    }
}
print join("\n==================\n",@list);

This is old, plain Perl style (and ugly, probably).

Srilesh ,Apr 23, 2010 at 18:30

I don't think pure regular expressions are what you want to use here (IMHO this might not even be parsable using regex).

Instead, build a small parser, similar to what's shown here: http://www.perlmonks.org/?node_id=308039 (see the answer by shotgunefx (Parson) on Nov 18, 2003 at 18:29 UTC)

UPDATE It seems it might be doable with a regex - I saw a reference to matching nested parentheses in Mastering Regular Expressions (that's available on Google Books and thus can be googled for if you don't have the book - see Chapter 5, section "Matching balanced sets of parentheses")

Borodin ,Mar 8, 2013 at 20:22

You're much better off using a state machine than a regex for this type of parsing.

> ,

Regular expressions are actually pretty bad for matching braces. Depending how deep you want to go, you could write a full grammar (which is a lot easier than it sounds!) for Parse::RecDescent . Or, if you just want to get the blocks, search through for opening '{' marks and closing '}', and just keep count of how many are open at any given time.

[Nov 21, 2019] Can the Perl debugger save the ReadLine history to a file?

Nov 21, 2019 | stackoverflow.com

Ask Question Asked 8 years, 5 months ago Active 6 years ago Viewed 941 times 10 2


eli ,Jun 7, 2018 at 14:13

I work quit a bit with lib ReadLine and the lib Perl Readline.

Yet, the Perl debugger refuses to save the session command line history.

Thus, each time I invoke the debugger I lose all of my previous history.

Does anyone know how to have the Perl debugger save, and hopefully, append session history similar to the bash HISTORYFILE ?

mirod ,Jun 22, 2011 at 10:31

The way I do this is by having the following line in my ~/.perldb file:

&parse_options("HistFile=$ENV{HOME}/.perldb.hist");

Debugger commands are then stored in ~/.perldb.hist and accessible across sessions.

ysth ,Jul 13, 2011 at 9:37

Add parse_options("TTY=/dev/stdin ReadLine=0"); to .perldb, then:
rlwrap -H .perl_history perl -d ...

mephinet ,Feb 21, 2012 at 12:37

$ export PERLDB_OPTS=HistFile=$HOME/.perldb.history

,

I did the following:

1) Created ~/.perldb , which did not exist previously.

2) Added &parse_options("HistFile=$ENV{HOME}/.perldb.hist"); from mirod's answer.

3) Added export PERLDB_OPTS=HistFile=$HOME/.perldb.history to ~/.bashrc from mephinet's answer.

4) Ran source .bashrc

5) Ran perl -d my program.pl , and got this warning/error

perldb: Must not source insecure rcfile /home/ics/.perldb.
        You or the superuser must be the owner, and it must not 
        be writable by anyone but its owner.

6) I protected ~/.perldb with owner rw chmod 700 ~/.perldb , and the error went away.

[Nov 18, 2019] How to convert list into string with quotes in python - Stack Overflow

Nov 18, 2019 | stackoverflow.com

How to convert list into string with quotes in python Ask Question Asked 6 years, 7 months ago Active 1 year, 2 months ago Viewed 9k times 4 0


alexis ,Apr 17, 2013 at 8:21

i have a list and want it as a string with quotes mylist = [1,2,3]

require O/P as myString = "'1','2','3'"

i tried mystring = '\',\''.join(mylist)

it gave me result as

mystring = "1','2','3"

first and last quotes (') are missing

Matthias ,Apr 17, 2013 at 8:53

This seems to be the only solution so far that isn't a hack...
>>> mylist = [1,2,3]
>>> ','.join("'{0}'".format(x) for x in mylist)
"'1','2','3'"

This can also be written more compactly as:

>>> ','.join(map("'{0}'".format, mylist))
"'1','2','3'"

jamylak ,Apr 17, 2013 at 8:36

as a simple hack, why don't you..
mystring = "'%s'" %"','".join(mylist)

wrap the result of your commands in quotes?

ashokadhikari ,Apr 17, 2013 at 8:24

you can do this as well
mylist = [1, 2, 3]
mystring = str(map(str, mylist)).strip("[]")

atomh33ls ,Apr 17, 2013 at 8:46

>>> mylist = [1,2,3]
>>> str([str(x) for x in mylist]).strip("[]")
"'1','2','3'"

> ,

OR regular repr :
>>> l=[1,2,3]
>>> ','.join(repr(str(i)) for i in l)
"'1','2','3'"
>>>

[Nov 18, 2019] Python enclose each word of a space separated string in quotes - Stack Overflow

Nov 18, 2019 | stackoverflow.com

Python: enclose each word of a space separated string in quotes Ask Question Asked 4 years, 1 month ago Active 1 year, 9 months ago Viewed 3k times 2


vaultah ,Sep 24, 2015 at 15:53

I have a string eg:
line="a sentence with a few words"

I want to convert the above in a string with each of the words in double quotes, eg:

 "a" "sentence" "with" "a" "few" "words"

Any suggestions?

mkc ,Sep 24, 2015 at 18:26

Split the line into words, wrap each word in quotes, then re-join:
' '.join('"{}"'.format(word) for word in line.split(' '))

Anand S Kumar ,Sep 24, 2015 at 15:54

Since you say -

I want to convert the above in a string with each of the words in double quotes

You can use the following regex -

>>> line="a sentence with a few words"
>>> import re
>>> re.sub(r'(\w+)',r'"\1"',line)
'"a" "sentence" "with" "a" "few" "words"'

This would take into consideration punctuations, etc as well (if that is really what you wanted) -

>>> line="a sentence with a few words. And, lots of punctuations!"
>>> re.sub(r'(\w+)',r'"\1"',line)
'"a" "sentence" "with" "a" "few" "words". "And", "lots" "of" "punctuations"!'

user4568737 user4568737

add a comment ,Aug 10, 2017 at 18:42
Or you can something simpler (more implementation but easier for beginners) by searching for each space in the quote then slice whatever between the spaces, add " before and after it then print it.
quote = "they stumble who run fast"
first_space = 0
last_space = quote.find(" ")
while last_space != -1:
    print("\"" + quote[first_space:last_space] + "\"")
    first_space = last_space + 1
    last_space = quote.find(" ",last_space + 1)

Above code will output for you the following:

"they"
"stumble"
"who"
"run"

Stephen Rauch ,Jan 28, 2018 at 3:29

The first answer missed an instance of the original quote. The last string/word "fast" was not printed. This solution will print the last string:
quote = "they stumble who run fast"

start = 0
location = quote.find(" ")

while location >=0:
    index_word = quote[start:location]
    print(index_word)

    start = location + 1
    location = quote.find(" ", location + 1)

#this runs outside the While Loop, will print the final word
index_word = quote[start:]
print(index_word)

This is the result:

they
stumble
who
run
fast

[Nov 15, 2019] Why are Unix system administrators still using Perl for scripting when they could use Python - Quora

Nov 15, 2019 | www.quora.com

Why are Unix system administrators still using Perl for scripting when they could use Python? Update Cancel

a OYLu d zEv ORPC b dRl y q nyXNY D AZ a eSr t gpl a yTipB d lH o xE g ookz H Tr Q voRm . iKPKM c YuOhH o M m HVViy Visualize Docker performance and usage in real time. Track Docker health and usage alongside custom metrics from your apps and services. Try Datadog for free. Learn More You dismissed this ad. The feedback you provide will help us show you more relevant content in the future. Undo Answer Wiki 12 Answers Joshua Day

Joshua Day , Currently developing reporting and testing tools for linux Updated Apr 26 · Author has 83 answers and 71k answer views

There are several reasons and ill try to name a few.

  1. Perl syntax and semantics closely resembles shell languages that are part of core Unix systems like sed, awk, and bash. Of these languages at least bash knowledge is required to administer a Unix system anyway.
  2. Perl was designed to replace or improve the shell languages in Unix/linux by combining all their best features into a single language whereby an administrator can write a complex script with a single language instead of 3 languages. It was essentially designed for Unix/linux system administration.
  3. Perl regular expressions (text manipulation) were modeled off of sed and then drastically improved upon to the extent that subsequent languages like python have borrowed the syntax because of just how powerful it is. This is infinitely powerful on a unix system because the entire OS is controlled using textual data and files. No other language ever devised has implemented regular expressions as gracefully as perl and that includes the beloved python. Only in perl is regex integrated with such natural syntax.
  4. Perl typically comes preinstalled on Unix and linux systems and is practically considered part of the collection of softwares that define such a system.
  5. Thousands of apps written for Unix and linux utilize the unique properties of this language to accomplish any number of tasks. A Unix/linux sysadmin must be somewhat familiar with perl to be effective at all. To remove the language would take considerable effort for most systems to the extent that it's not practical.. Therefore with regard to this environment Perl will remain for years to come.
  6. Perl's module archive called CPAN already contains a massive quantity of modules geared directly for unix systems. If you use Perl for your administration tasks you can capitalize on these modules. These are not newly written and untested modules. These libraries have been controlling Unix systems for 20 years reliably and the pinnacle of stability in Unix systems running across the world.
  7. Perl is particularly good at glueing other software together. It can take the output of one application and manipulate it into a format that is easily consumable by another, mostly due to its simplistic text manipulation syntax. This has made Perl the number 1 glue language in the world. There are millions of softwares around the world that are talking to each other even though they were not designed to do so. This is in large part because of Perl. This particular niche will probably decline as standardization of interchange formats and APIs improves but it will never go away.

I hope this helps you understand why perl is so prominent for Unix administrators. These features may not seem so obviously valuable on windows systems and the like. However on Unix systems this language comes alive like no other.

[Nov 15, 2019] Python Displaces C++ In TIOBE Index Top 3 - Slashdot

Nov 15, 2019 | developers.slashdot.org

Posted by EditorDavid on Saturday September 08, 2018 @03:34PM from the newer-kid-on-the-block dept. InfoWorld described the move as a "breakthrough": As expected, Python has climbed into the Top 3 of the Tiobe index of language popularity, achieving that milestone for the first time ever in the September 2018 edition of the index. With a rating of 7.653 percent, Python placed third behind first-place Java, which had a rating of 17.436 percent, and second-place C, rated at 15.447. Python displaced C++, which finished third last month and took fourth place this month, with a rating of 7.394 percent...

Python also has been scoring high in two other language rankings:

- The PyPL Popularity of Programming Language index, where it ranked No. 1 this month , as it has done before, and has had the most growth in the past five years.

- The RedMonk Programming Language Rankings, where Python again placed third .
Tiobe notes that Python's arrival in the top 3 "really took a long time," since it first entered their chart at the beginning of the 1990s. But today, "It is already the first choice at universities (for all kinds of subjects for which programming is demanded) and is now also conquering the industrial world." In February Tiobe also added a new programming language to their index: SQL. (Since "SQL appears to be Turing complete.")

"Other interesting moves this month are: Rust jumps from #36 to #31, Groovy from #44 to #34 and Julia from #50 to #39."
0 Share Facebook Twitter LinkedIn Reddit Share this with your friends! From To Compose your message Share share image Python Displaces C++ In TIOBE Index Top 3 https://developers.slashdot.org/story/18/09/08/1722213/python-displaces-c-in-tiobe-index-top-3

programming stats python Related Links Wikipedia Seeks Photos of 20 Million Artifacts Lost in Brazil Museum Fire Interviews: Guido van Rossum Answers Your Questions IEEE Spectrum Declares Python The #1 Programming Language Is C++ a 'Really Terrible Language'? Python Language Founder Steps Down Beta Release Nears For BeOS-inspired Open Source OS Haiku This discussion has been archived. No new comments can be posted. Python Displaces C++ In TIOBE Index Top 3 54 More Login Python Displaces C++ In TIOBE Index Top 3 Comments Filter: All Insightful Informative Interesting Funny

The Fine Print: The following comments are owned by whoever posted them. We are not responsible for them in any way.


Anonymous Coward writes:

Congratulations to the Python maintainers ( Score: 1 )

... but if they hadn't handled the Python 2/3 fork so clumsily, this might have happened years ago.

raymorris ( 2726007 ) , Saturday September 08, 2018 @03:58PM ( #57276974 ) Journal
Losing backwards compatibility with point releases ( Score: 5 , Interesting)

Never mind Python 2 vs 3; one major reason I shy away from Python is the incompatibility in point releases. I'd see "requires Python 2.6â and see that I have Python 2.7 so it should be fine, right? Nope, code written for 2.6 won't run under Python 2.7. It needs to be EXACTLY 2.6.

It's at this point that some Python fanboi gets really upset and starts screaming about how that's now problem, with Python you set up separate virtual environments for each script, so that each one can have exactly the version of Python it is written for, with exactly the version of each library. When there is some bug or security issue you then hope that there is a patch for each, and deal with all that. (As opposed to every other peice of software in the world, which you simply upgrade to the latest version to get all the latest fixes). Yes, you CAN deal with that problem, it's possible, in most cases. You shouldn't have to. Every other language does some simple things to maintain backward compatibility in point releases (and mostly in major releases too).

Also the fact that most languages use every day and have used for decades use braces for blocks means my eyes and mind are very much trained for that. Braces aren't necessarily BETTER than just using indentation, but it's kinda like building a car which uses the pedals to steer and a hand stick for the gas. It's not necessarily inherently better or worse, but it would be almost undriveable for an experienced driver with decades of muscle memory in normal cars. Python's seemingly random choices on these things make it feel like using your feet to steer a car. There should be compelling reasons before you break long-established conventions and Python seems to prefer to break conventions just to be different. It seems the Python team is a bit like Berstein in that way. It's really annoying.

slack_justyb ( 862874 ) , Saturday September 08, 2018 @05:00PM ( #57277244 )
Re:Losing backwards compatibility with point relea ( Score: 5 , Informative)

Python 2.7. It needs to be EXACTLY 2.6

Yeah, just FYI Python 2.7 is in a way its own thing. Different from the 2.x and different from the 3.x series. 2.6 is a no holds barred pure 2.x whereas 2.7 is a mixture of 2.x and 3.x features. So if you want to compare point releases, best to try that with the 3.x series. Also, if you're using something that requires the 2.x series, you shouldn't use that unless it is absolutely critical with zero replacements.

You shouldn't have to. Every other language does some simple things to maintain backward compatibility in point releases (and mostly in major releases too).

Again see argument about 3.x, but yeah not every language does this. Java 8/9 transition breaks things. ASP.Net to ASP.Net core breaks things along the way. I'm interested in what languages you have in mind, because I know quite a few languages that do maintain backwards compatibility (ish). For example, C++ pre and post namespace breaks fstreams in programs, but compilers provide flags to override that, so it depends on what you mean by breaking. Does it count if the compiler by default breaks, but providing flags fixes it? Because if your definition means including flags break compatibility, then oooh boy are there a a shit ton of broken languages.

Also the fact that most languages use every day and have used for decades use braces for blocks means my eyes and mind are very much trained for that

Yeah, it's clear that you've never used a positional programming language. I guess it'll be a sign of my age, but buddy, program COBOL or RPG on punch cards and let me know about that curly brace issue you're having. Positional and indentation has been used way, way, way, longer than curly braces. That's not me knocking on the curly braces, I love my C/C++ folks out there! But I hate to tell you C and C-style is pretty recent in the timeline of all things computer.

raymorris ( 2726007 ) writes:
Depends on the results / message, actually ( Score: 3 , Insightful)

> C++ pre and post namespace breaks fstreams in programs, but compilers provide flags to override that, so it depends on what you mean by breaking. Does it count if the compiler by default breaks, but providing flags fixes it?

If it results in weird runtime errors, that's definitely a problem.
If the compiler I'm using gives the message "incompatible use of fstream, try '-fstreamcompat' flag", that's no big deal.

raymorris ( 2726007 ) writes:
Also deprecation warnings ( Score: 2 )

On a similar note, if something is marked deprecated long before it's removed, that matters. Five years of compiler/interpreter warnings saying "deprecated use of function in null context on line #47" gives plenty of opportunity. To fix it. From the bit of Python I've worked with, the recommended method on Friday completely stops working on Monday.

shutdown -p now ( 807394 ) writes:
Re: ( Score: 2 )

That's plainly not true - Python follows the established deprecate-first-remove-next cycle. This is readily obvious when you look at the changelogs. For example, from the 2.6 changelog:

The threading module API is being changed to use properties such as daemon instead of setDaemon() and isDaemon() methods, and some methods have been renamed to use underscores instead of camel-case; for example, the activeCount() method is renamed to active_count(). Both the 2.6 and 3.0 versions of the module support the same properties and renamed methods, but don't remove the old methods. No date has been set for the deprecation of the old APIs in Python 3.x; the old APIs won't be removed in any 2.x version.

For another example, the ability to throw strings (rather than BaseException-derived objects) was deprecated in 2.3 (2003) and finally removed in 2.6 (2008).

For comparison, in the C++ land, dynamic exception specifications were deprecated in C++11, and removed in C++17. So the time scale is comparable.

raymorris ( 2726007 ) writes:
That's nice they deprecated something ( Score: 2 )

That's great that they deprecate something on some occasions.
MY experience with the Python I run is that one version gives no warning, going up one point release throws multiple fatal errors.

> This is readily obvious when you look at the changelogs

Maybe that's the thing - one has read the changelogs to see what is deprecated, as opposed to getting a clear deprecation warning from the interpreter/compiler like you would with C, Perl, and other languages?

It's possible that a Python expert might be able to

shutdown -p now ( 807394 ) writes:
Re: ( Score: 2 )

MY experience with the Python I run is that one version gives no warning, going up one point release throws multiple fatal errors.

Can you give an example? I'm just not aware of any, and it makes me suspect that what you were running into was an issue in a third-party library (some of which do indeed have a cowboy attitude towards breaking changes - but that's common across all languages).

Maybe that's the thing - one has read the changelogs to see what is deprecated, as opposed to getting a clear deprecation warning from the interpreter/compiler like you would with C, Perl, and other languages?

Like this [python.org]?

And I have never, ever seen a deprecation warning in C or C++. You have to read the change sections for new standards to see what was deprecated or removed.

raymorris ( 2726007 ) writes:
Default warns, unless you turn it off in CFLAGS ( Score: 2 )

> And I have never, ever seen a deprecation warning in C or C++. You have to read the change sections for new standards to see what was deprecated or removed.

The default with gcc is to warn about deprecation.
You can turn the warnings off by setting the CFLAGS environment variable to include -Wno-deprecated, which you can do in your .bashrc oe wherever. What's most often recommended is -Wall to show all warnings of all types.

johannesg ( 664142 ) writes:
Re: ( Score: 3 )

For example, C++ pre and post namespace breaks fstreams in programs, but compilers provide flags to override that

Dude, that was in 1990, back before there even was a standard C++. And I very much doubt those flags still exist today.

program COBOL or RPG on punch cards and let me know about that curly brace issue you're having

You seem to have forgotten how that really worked in your old age though. Punch cards had columns with specific functions assigned to them, so yes, of course you would have to skip certain columns on occasion. That was not indentation, though. You didn't have indentation; moving your holes by one position or one column meant the machine would interpret your instruction as something else ent

sjames ( 1099 ) writes:
Re: ( Score: 3 )

That must be an odd package. I have literally NEVER seen that with anything I have wanted to use, including my own pre-2.7.x software.

fluffernutter ( 1411889 ) writes:
Warrior ( Score: 1 )

Real code warriors don't need static types. If a variable is so badly named that the type is not clear, use type().

PhrostyMcByte ( 589271 ) writes:
Re: ( Score: 2 )

If a variable is so badly named that the type is not clear

Never fear, I've brought my LPCWSTR.

jd ( 1658 ) writes:
Re: Warrior ( Score: 3 , Insightful)

Static typing isn't just about clarity to the programmer. In strict typing languages, the rule is to use the type that matches the range that actually applies. This is to help testing (something coders should not ignore), automated validation, compilation (a compiler can choose sensible values, optimise the code, etc etc etc) and maintainers (a clear variable name won't tell anyone if a variable's range can be extended without impacting the compiled code).

Besides, I've looked at Python code. I'm not convinc

serviscope_minor ( 664417 ) writes:
Re: ( Score: 2 )

Real code warriors don't need static types. If a variable is so badly named that the type is not clear, use type().

pfaaah. Real programmers don't need names. If the type of a variable is not obvious from context, get another job.

Undead Waffle ( 1447615 ) writes:
Re: ( Score: 2 )

Type annotations and docstrings help with the whole lack of type declaration thing. Of course that requires discipline, which is in short supply from my experience. If you can force your developers to run pylint that will at least complain when they don't have docstrings.

Aighearach ( 97333 ) writes:
Re: ( Score: 2 )

You definitely never had to debug student's code.

Is that even a thing?

mi ( 197448 ) writes: < slashdot-2017q4@virtual-estates.net > on Saturday September 08, 2018 @03:42PM ( #57276880 ) Homepage Journal
If Java is the first... ( Score: 5 , Insightful)

behind first-place Java

Whatever the list, if Java is in the first place, there is no honor in being anywhere near the top.

jd ( 1658 ) writes: < imipak@yahoo.cUUUom minus threevowels > on Saturday September 08, 2018 @04:18PM ( #57277066 ) Homepage Journal
Re: If Java is the first... ( Score: 4 , Insightful)

The list is compiled from a restricted pool and lists popularity.

That may mean a vendor throwing out ten individually packaged Python scripts counts as ten sources with one C program of equalling counts as one. If that's the case, Python would be ten times as popular in the stats whilst being equally popular in practice.

So if Python needed ten times as many modules to be as versatile, it would seem popular whilst only being frustrating.

The fact is, we don't know their methodology. We don't know if they're weighting results in any way to factor in workarounds and hacks due to language deficiency that might show up as extra use.

We do know they don't factor in defect density, complexity or anything else like that as they do say that much. So are ten published attempts at a working program worth ten times one attempt in a language that makes it easy first time? We will never know.

nten ( 709128 ) writes:
vba ( Score: 2 )

I find java in an uncanny valley. Its still a few times slower than c++ for the sort of stuff I do but it isn't enough quicker to develop than c++ to be worth that hit. Python is far slower than java even using numpy but its so easy to develop in that it is worth the gamble that it will be fast enough. And the rewrite in c++ will go quickly even if it isn't. The title is because VBA is 11x faster than numpy at small dense matricies and almost as easy to develop in.

phantomfive ( 622387 ) writes:
Re: vba ( Score: 2 )

Java is useful because you can throw a team of lowskill developers at it and they won't mess things up beyond the point of unmaintanability. It will be a pain to maintain, sure, but the same developers using C would make memory errors that push things beyond hopeless, and if they were using Python or JavaScript the types would become more and more jumbled as the size of the program increases that no one would be able to understand it and things would start breaking more and more. Java enforces a minimal lev

angel'o'sphere ( 80593 ) writes:
Re: ( Score: 2 )

Luckily I usually have high skilled developers in my Java projects :P

phantomfive ( 622387 ) writes:
Re: vba ( Score: 2 )

Then your codebase is better.

Tough Love ( 215404 ) writes:
Re: ( Score: 3 )

Tiobe is utter crap. Javascript (barf) is by far the most popular programming language today and Tiobe puts it in 8th place, behind Visual Basic.

ljw1004 ( 764174 ) writes:
Re: If Java is the first... ( Score: 2 )

From the same article - Visual Basic overtakes C#, PHP and Javascript...

El Cubano ( 631386 ) , Saturday September 08, 2018 @03:59PM ( #57276982 )
Love Python ( Score: 3 )

Tiobe notes that Python's arrival in the top 3 "really took a long time," since it first entered their chart at the beginning of the 1990s. But today, "It is already the first choice at universities (for all kinds of subjects for which programming is demanded)

Undergraduate was all C/C++ for me then I ended up at a graduate school where everything was Java. I disliked it so much that I decided to find an alternative and teach myself. I found Python and loved it. I still love it. You can't find anything better for both heavy duty programming and quick and dirty scripting. It's versatility makes It like the Linux of programming languages.

Tough Love ( 215404 ) , Saturday September 08, 2018 @05:02PM ( #57277252 )
Re:Love Python ( Score: 4 , Insightful)

I found Python and loved it. I still love it. You can't find anything better for both heavy duty programming...

What? Python is hopelessly inefficient for heavy duty programming, unless you happen to be doing something that is mainly handled by a Python library, written in C. Python's interface to C disgusting, so if you have a lot of small operations handled by a C library, you will get pathetic performance.

sjames ( 1099 ) writes:
Re: ( Score: 2 )

It really isn't. There are some apps that actually need something faster and a lot of apps that don't. It really doesn't help if a faster executable ends up waiting for I./O.

Tough Love ( 215404 ) writes:
Re: ( Score: 3 )

It really isn't.

It really is [debian.net] and you blathering about what you don't know does not change that fact. (Python 14 minutes vs C++ 8..24 seconds for N-Body simulation.)

sjames ( 1099 ) writes:
Re: ( Score: 2 )

Well, since 99.99999999999% of all software run by literally everybody is an n-body simulation....

That would be an example of the "some apps" I spoke of. I note that Intel Fortran was at the top of the list (not surprising). Would ifort be your first choice if you were writing a text editor or a tar file splitter? How about an smtp daemon?

I sure hope not.

Tough Love ( 215404 ) writes:
Re: ( Score: 2 )

Well, since 99.99999999999% of all software run by literally everybody is an n-body simulation..

Explaining the concept of "compute intensive" to you makes me feel more stupid. Check out any of the compute intensive Python benchmarks. Consider not waving your ignorance around quite so much.

sjames ( 1099 ) writes:
Re: ( Score: 2 )

Having actually built a cluster that was in the top 500 for a while, I am well acquainted with compute intensive applications. I am also aware that compute intensive is a subset of "heavy duty" programming which is a subset of general programming.

Now, pull your head out of your ass and look around, you might learn something. And while you're at it, consider working on your social skills.

Tough Love ( 215404 ) writes:
Re: ( Score: 2 )

Either you understand that Python is crap for compute intensive work, or you are lying about building a cluster. Or you just connected the cables, more like it, and really don't have a clue about how to use it.

sjames ( 1099 ) writes:
Re: ( Score: 2 )

I do understand that python isn't the right choice for compute intensive work. With the exception that if it is great for doing setup for something in FORTRAN or C that does the heavy lifting.

I am certain that YOU don't understand that compute intensive work is a small fraction of what is done on computers. For example, I/O intensive work doesn't really care if it is Python or FORTRAN that is waiting for I/O to complete. There is a reason people don't drive a top fuel dragster to work.

If you meant compute i

Tough Love ( 215404 ) writes:
Re: ( Score: 2 )

I am certain that YOU don't understand that compute intensive work is a small fraction of what is done on computers.

First, you have no idea what I do or do not understand because you find yourself way too entertained by your own blather, and second, computers are used more for browsing than any other single task these days, and wasteful use of the CPU translates into perceptible lag. Playing media is very CPU intensive. You don't write those things in Python because Python sucks for efficiency. My point.

Yes, I had you figured, you're a sysadmin with delusions about being a dev. Seen way too many of those. They tend to ta

sjames ( 1099 ) writes:
Re: ( Score: 2 )

I draw my conclusions from what you have written in this thread. You see one screw and think NOTHING is a nail.

You don't write a video codec in Python, but it's a great choice for handling the UI and feeding the stream to the codec.

You have much to learn.

Tough Love ( 215404 ) writes:
Re: ( Score: 2 )

As I said, "Python is hopelessly inefficient for heavy duty programming". WTF are you blathering on about. Fresh air is good for you, maybe get out of your basement more.

sjames ( 1099 ) writes:
Re: ( Score: 2 )

"Heavy Duty " != "compute intensive". Say what you mean or don't complain when people disagree.

Tough Love ( 215404 ) writes:
Re: ( Score: 2 )

OK, you have your own private definition of terminology. Enjoy life in your own private universe.

sjames ( 1099 ) writes:
Re: ( Score: 2 )

I made a couple attempts to clarify terminology but you were too busy looking for an excuse to make an ass of yourself to notice.

K. S. Kyosuke ( 729550 ) writes:
Re: ( Score: 2 )

It really is [debian.net] and you blathering about what you don't know does not change that fact. (Python 14 minutes vs C++ 8..24 seconds for N-Body simulation.)

I've just run it on my machine. C++: 2.3 seconds, Python: 22 seconds. That's for straightforward mathy Python against C++ code with vector instrinsics. Concerning C++ code without manual vectorization, it's 4 seconds against 22. Not terribly bad, I'd say. Not to mention that this isn't the kind of code that would be typical for a larger application.

Tough Love ( 215404 ) , Saturday September 08, 2018 @09:44PM ( #57278110 )
Re:Love Python ( Score: 3 )

5x+ penalty just for writing the code in Python, you call it not terribly bad? So this is how Python fans think.

angel'o'sphere ( 80593 ) writes: < angelo...schneider@@@oomentor...de > on Saturday September 08, 2018 @10:19PM ( #57278224 ) Journal
Re:Love Python ( Score: 2 )

The first python program I wrote was a test for a job interview.
It involved downloading meteorologic data from the internet.
Analyzing it, creating a kind of summary and using a graph plotting library to display a graph (generate a *.png)

It would not have been noticeable faster if I had written it in C++, because ... you know: downloading via a network.

K. S. Kyosuke ( 729550 ) writes:
Re: ( Score: 2 )

I'm pretty sure you get another 3x penalty for not writing in assembly, too. So this is how C++ fans think.

Tough Love ( 215404 ) writes:
Re: ( Score: 2 )

I'm pretty sure you get another 3x penalty for not writing in assembly, too...

You are sure of that, are you? I bet you have never coded in assembly yourself, or looked at the assembly that gcc puts out in O3.

K. S. Kyosuke ( 729550 ) writes:
Re: ( Score: 2 )

I did, for 6502, 8051 and 8086. And I've seen GCC's output.

Tough Love ( 215404 ) writes:
Re: ( Score: 2 )

Then you looked at it without understanding it. Do you seriously think you can out-optimize gcc's code generator? Do you even know how to use LEA for arithmetic?

K. S. Kyosuke ( 729550 ) writes:
Re: ( Score: 2 )

LEA is not going to help you much in a small fixed-size n-body kernel. That's going to be mostly unrolled AVX code.

dwpro ( 520418 ) writes:
Re: ( Score: 2 )

Have you worked on or have an example of a large, complex application in Python? I'd like to see how it's organized, seems like it'd be a nightmare.

Paul Carver ( 4555 ) writes:
Re: ( Score: 2 )

https://github.com/openstack/n... [github.com]

flargleblarg ( 685368 ) writes:
Re: ( Score: 2 )

Undergraduate was all C/C++ for me [...]

And I believe you! -- because...

It's versatility makes It [...]

...evidentally you forgot to take grammar class. ;)

Anonymous Coward writes:
I'm looking for a language ... ( Score: 1 )

Is there a programming language out there, that is as fast as C++ or even C, has a proper strict type system (no duck typing, nothing like Python or JS), fast garbage collection (no fuckin' auto_pointer worst-of-both-worlds), is elegant and emergent (so very powerful for its simplicity), and doesn't require an advanced degree in computer sciences to do simple things (Hello Haskell!).
Of course with key libraries being available for it. (The equivalent of a standard library, Vulcan, a simple GUI widget toolki

Tablizer ( 95088 ) writes:
Re: ( Score: 1 )

It's called "IwannaPony", and you are NOT going to get one. You are asking too much.

CQDX ( 2720013 ) writes:
Qt (with C++) ( Score: 2 )

I really like the Qt framework. It's well done, well documented and well supported. Sure it's C++ so it doesn't meet your need of finding a new language but the API is pretty clean and simple so that you can avoid the complications and ugliness of C++ in most cases. If you need to though, it's all right there so you don't give up the additional power if you need it. The Python version is good too and very similar to the C++ version so it's not hard to switch between languages as your needs change.

serviscope_minor ( 664417 ) writes:
Re: ( Score: 2 )

Is there a programming language out there, that is as fast as C++ or even C, has a proper strict type system (no duck typing, nothing like Python or JS), fast garbage collection

No.

Neither will there be. There's always a penalty for garbage collection.

rkcth ( 262028 ) writes:
Re: ( Score: 1 )

Is there a programming language out there, that is as fast as C++ or even C, has a proper strict type system (no duck typing, nothing like Python or JS), fast garbage collection

No.

Neither will there be. There's always a penalty for garbage collection.

I think go is the closest to your requested feature list.

serviscope_minor ( 664417 ) writes:
Re: ( Score: 2 )

I think go is the closest to your requested feature list.

The GP, not mine. And yuck, no thanks. Go just seems, well, deeply mediocre in many places. It's like someone pdated C, ignoreing the last 40 years of language developments.

Sure I can program without generics, I'm at a loss to see why I'd want to though.

Spacelem ( 189863 ) writes:
Re: ( Score: 2 )

Julia? It ticks quite a few of those boxes.

Its performance is good enough that I'm able to drop C++ (I'm a mathematical modeller), it's amazing at multidimensional array manipulation, and its typing system is really good. It just feels nice to program in. Bonus, one of the inspirations was Lisp, so it's got good metaprogramming. Also it's free software, made by people at MIT, so your conscience can remain appeased.

It's still a young language, but libraries are being built for it at an impressive rate, and i

TechyImmigrant ( 175943 ) writes:
Re: ( Score: 2 )

I'm holding out for Jai.

jd ( 1658 ) writes:
I would advise against any university ( Score: 2 )

That advocated a language. Languages shift faster than sand on speed. Universities should teach logic, reasoning, methodology, good practices and programming technique.

Languages should be for the purpose of example only. Universities should teach programming, not Java, software engineering, not Python. Java and Python should be in there, yes, along with Perl, C and Ada. Syntax is just sugar over the semantics. Learn the semantics well and the syntax is irrelevant. You want universities to teach kids how to

Tablizer ( 95088 ) writes:
Re: ( Score: 1 )

when Cobol and Fortran were the in thing. Last forever, they thought.

Any evidence most universities believed that? (They are still around and relatively common, by the way.)

Universities have to pick something to program lesson projects in, and selecting language(s) common in the current market helps student job prospects. I suggest STEM students be required to learn at least one compiled/strong-typed language, and one script/dynamic language.

TechyImmigrant ( 175943 ) writes:
Re: ( Score: 2 )

My university (Manchester University, UK) certainly didn't pick a language.

We studied many languages, compiler design, formal semantics and a boatload of other computer science things but at no point did they try to teach me a programming language. In fact at induction they said explicitly that they expected us all to know how to program before we arrived.

That was 30 years ago. Things might have changed.

petermgreen ( 876956 ) writes:
Re: ( Score: 2 )

(note: this is a UK perspective, other places may vary)

Universities have to work with the students they can get.

I think you and your co-students were lucky to catch the height of the 80s microcomputer boom, the time when computers booted into BASIC, when using a computer pretty much meant programming it.

Then the windows PCs with their pre-canned applications and no obvious programming language swept in. Leaning to program now meant not just finding a suitible book, it often meant buying the programming lang

Tablizer ( 95088 ) writes:
Re: ( Score: 1 )

My university...didn't pick a language.

Didn't you have projects that involved turning in your code to the teacher/graders? The graders don't want to see every which language. Multi-lingual graders are more expensive. Most colleges dictate a narrow set of languages for such projects.

TechyImmigrant ( 175943 ) writes:
Re: ( Score: 2 )

My university...didn't pick a language.

Didn't you have projects that involved turning in your code to the teacher/graders? The graders don't want to see every which language. Multi-lingual graders are more expensive. Most colleges dictate a narrow set of languages for such projects.

Yes, but it was hardly narrow. We had homework to hand in using a variety of languages, depending on the course. Pascal tended to be used for general algorithm stuff. But Smalltalk, Prolog, ML and other usual suspects were used when they made sense for the course. You were supposed to leave with a CompSci degree where you understood the theory of languages more than the details of specific languages. Usually, for project work, you were free to choose your language and would be expected to justify the reason

Tablizer ( 95088 ) writes:
Re: ( Score: 1 )

I don't remember herds of graders. That's what postgrads are for.

Maybe the graduation % was too low at my U.

pauljlucas ( 529435 ) writes:
Re: ( Score: 2 )

If you want to teach semantics, use either Smalltalk or Scheme. You can teach the syntax for either in five minutes.

pilaftank ( 1096645 ) writes:
Go Groovy! ( Score: 1 )

Groovy from #44 to #34

That's a pretty big jump. Groovy is a well-thought-out language and nicely facilitates writing clean, readable, compact code (especially compared to Java). However, it needs a better framework than Grails (85% really good convention over configuration stuff but 15% convoluted j2ee era framework stuff).

Tablizer ( 95088 ) writes:
Hackers love TC ( Score: 1 )

[SQL added to list] since "SQL appears to be Turing complete."

Not sure that's a good thing.

cats-paw ( 34890 ) writes:
python3 for full application development. wtf? ( Score: 2 , Interesting)

Can someone explain to me why using a dynamically typed language is a good idea for "big" applications ?

Python is subject to all sorts of really horrendous bugs that would not happen in a compiled, type-checked language.

For example if you are accessing an undefined variable in the else branch of an if statement, you won't know it's undefined unless that branch is taken. which means if it's something like a rarely occurring error condition it's kind of annoying. yes you can figure it out by writing enough t

Frankablu ( 812139 ) writes:
Re: ( Score: 1 )

It's really simple, Writing an application in Python is x3 quicker than writing it in C/C++/Java, etc... That means you either get to market 3x faster or only need 1/3 the number of programmers. Everything else is completely and utterly irrelevant. "you won't know it's undefined unless that branch is taken" The code linter built into your Python IDE, will tell you about it.

dwpro ( 520418 ) writes: < dgeller777@@@gmail...com > on Sunday September 09, 2018 @05:06AM ( #57278852 )
Re:python3 for full application development. wtf? ( Score: 2 )

Hogwash. Even if x3 were true, Dev is roughly 20-40% of overall software cost. Unless you're arguing that every aspect of coding is reliably 3x faster in Python. Given the value of strong typing when refactoring, I'd wager python is not even competitive price wise past the proof of concept/one-off script scenario.

Frankablu ( 812139 ) , Sunday September 09, 2018 @07:52AM ( #57279148 ) Journal
Re:python3 for full application development. wtf? ( Score: 1 )

I'm am arguing that because it's true. There isn't much benefit to strong typing when refactoring but the benefits of duck typing when it comes to unit testing are quite significant. I've done commercial software development in strong and weakly typed languages before. The benefits of things like "strong typing" are generally not that much. If you are on board with the whole agile bandwagon and writing unit tests and all that. You would be much better off with Python's significally better unit testing facilities than strong typing.

dwpro ( 520418 ) writes:
Re: ( Score: 2 )

I'm at least in the caravan trailing the agile/unit-test bandwagon, but those are orthogonal to typing (and being explicit generally). Looking at a method signature and knowing that it requires a decimal and enumeration of a given type is more than a run-time check; it provides information about the intent, limitations, and discoverability options. There are very real trade-offs for the speed and flexibility of a language like Python, and my view is that it's more jet-fuel than solar power.

Frankablu ( 812139 ) writes:
Re: ( Score: 1 )

If you are using a good IDE "provides information about the intent, limitations, and discoverability options" can all be found out with a couple of key strokes (git blame, find all usages, pylint, etc..). So putting that information into the language explicitly is an obsolete and backwards way of going about things :p The job of the compiler in Python has just been redistributed elsewhere. It's different but there are many ways to solve the same problems.

sjames ( 1099 ) writes:
Re: ( Score: 2 )

Six of one, half dozen of the other. The Python program will be smaller for the same functionality and it won't have buffer overflows and memory leaks The C program will run faster (unless it has to wait on I/O) and will check for variables used before assignment.

munch117 ( 214551 ) , Sunday September 09, 2018 @03:40AM ( #57278724 )
Re:python3 for full application development. wtf? ( Score: 4 , Interesting)

Using an undefined variable in Python triggers an exception, and you get a traceback. In a larger program you will normally have a system for capturing and storing such tracebacks for analysis, and with the traceback in hand, it's typically a very simple fix.

In C++ you get an incorrect value created by default-initialisation (or maybe undefined behaviour): the program hobbles along as best it can, and you may never find the problem. You just see your program behaving strangely sometimes, and as the program gets larger, those strange behaviours accumulate.

Python is subject to all sorts of really horrendous bugs that would not happen in a compiled, type-checked language.

Horrendous is not the right word. Bugs that come with tracebacks are simple bugs. Zen#10: "Errors should never pass silently" is exactly what you want in large-scale programming.

shutdown -p now ( 807394 ) writes:
Re: ( Score: 2 )

By "undefined variable" I think he means undeclared (or, in Python, unassigned in the proper scope, since locals are declared implicitly).

donbudge ( 4988829 ) writes:
Re: ( Score: 1 )

Writing big applications in Java/C++ takes too long. And then managements decide to avoid 'custom code' in favor of 'standard' vendor tools where you can drag and drop to build parts of the 'big' application. This applies to ETL, reporting, messaging to name a few. With Python, the development cycle shortens and you can still stick to writing code instead of dealing with vendor binaries, lock-ins, licensing etc. Python with strong emphasis on unit tests, coupled with plugging in C/C++ where necessary f

nashv ( 1479253 ) writes:
Re: ( Score: 2 )

If you are really really asinine about strong typing, you can declare types in Python https://medium.com/@ageitgey/l... [medium.com]

SuperKendall ( 25149 ) , Saturday September 08, 2018 @09:51PM ( #57278124 )
Hello Machine Learning ( Score: 4 , Interesting)

I think what has really propelled Python into a higher rank is machine learning, where it is simply the de-facto language of choice by quite a margin.

I have to admit I am impressed with the progress it has made; of many recent CS grads I've talked to it seemed to be the favorite language...

I have to admit that over the years I've not really enjoyed Python much myself in the on and off again times I've used it, for me the spaces as indent levels maybe get too close to the meaningful whitespace of Fortran... I guess modern programmers do not have this hangup. :-)

So good work Python, a well deserved ascent!

shutdown -p now ( 807394 ) writes:
Re: ( Score: 2 )

It's not just ML, but data science in general.

And the other big thing was many universities switching to it from Java for their CS courses.

SuperKendall ( 25149 ) writes:
Re: ( Score: 2 )

That's a great point, and to be honest that is probably a better language for learning than Java... it would also explain why newer CS grads all like it more now.

The only downside is that most jobs are still using Java or something besides python... but probably it means we'll se more python used in businesses I guess. That usually ends up following eventually.

CustomSolvers2 ( 4118921 ) , Sunday September 09, 2018 @03:06AM ( #57278678 ) Homepage
Python is very newcomer-friendly ( Score: 2 )

I have been sporadically using Python for some years already and never really liked it. Note that most of my experience is focused on C-based and strongly-typed programming languages. Recently, I have been spending some time on a Python project and have realised about its (newbie) friendliness.

I still don't quite understand the reason for all the tabs/spaces problems, consider it too slow, don't like the systematic need of relying on external resources and I will certainly continue using other languages before it. But I do understand now why newbies or those performing relatively small developments or those wanting to rely on some of the associated resources (although I don't like being systematically forced to include external dependencies, I do recognise that Python deals with these aspects quite gracefully and that there are many interesting libraries) might prefer Python. It is one of the most intuitive programming languages which I have ever used, at least from what seems a newcomer perspective (e.g., same command performing what are intuitively seen as similar actions).

zmooc ( 33175 ) writes:
Too simple ( Score: 2 )

I don't think this should be about lines of code written. A more interesting approach would be to also count all dependencies, counting things like libc a gazillion times. Even more interesting would be to count what's actually executed.

casperghst42 ( 580449 ) writes:
RIght... ( Score: 1 )

Not typesafe, and no switch/case ... what ever rocks your boat.

vux984 ( 928602 ) writes:
Re: ( Score: 2 )

"Or any of the games in my library, which all appear to be C or C++, with a few C#."

From what I've seen a lot of the core engine stuff is C/C++; but a lot of the UI, AI, and "mod support" stuff is commonly done in Python and Lua.

Personally, I disagree with semantic whitespace so I don't like python. (I think its the editors job to handle pretty formatting to reflect the structure, rather than the programmers job to define structure with pretty formatting.) But I can see why python would be a good learning l

angel'o'sphere ( 80593 ) writes:
Re: ( Score: 3 )

Eve Online is mostly written in Python, client and server.
It is the MMO game with the most concurrent users online at any time of the day.

Speed is not their problem.

vux984 ( 928602 ) writes:
Re: ( Score: 2 )

'bad habits' ? what sort do you think?

I sort of see it as the opposite... semantic whitespace teaches mostly good habits, its just fucking irritating to maintain, and to work with snippets and code fragments etc.

But its highly readable, and pretty straightforward, and i don't see anything wrong with it as a beginning/educational language; for teaching flow control, algorithms, structured/modular programming, and so on.

[Nov 15, 2019] Go versus Python 3 -- fastest programs

Nov 15, 2019 | pages.debian.net
Back in April 2010, Russ Cox charitably suggested that only fannkuch-redux, fasta, k-nucleotide, mandlebrot, nbody, reverse-complement and spectral-norm were close to fair comparisons. As someone who implemented programming languages, his interest was "measuring the quality of the generated code when both compilers are presented with what amounts to the same program."

Differences in approach - to memory management, parallel programming, regex, arbitrary precision arithmetic, implementation technique - don't fit in that kind-of fair comparison -- but we still have to deal with them.

These are only the fastest programs. There may be additional measurements for programs which seem more-like a fair comparison to you. Always look at the source code.

mandelbrot
source secs mem gz busy cpu load
Go 5.47 31,088 905 21.77 100% 99% 99% 99%
Python 3 259.50 48,192 688 1,036.70 100% 100% 100% 100%
spectral-norm
source secs mem gz busy cpu load
Go 3.94 2,740 548 15.74 100% 100% 100% 100%
Python 3 169.87 49,188 417 675.02 100% 99% 99% 99%
n-body
source secs mem gz busy cpu load
Go 21.25 1,588 1310 21.48 0% 1% 100% 0%
Python 3 865.18 8,176 1196 874.96 2% 20% 79% 0%
fasta
source secs mem gz busy cpu load
Go 2.08 3,560 1358 5.61 80% 37% 76% 78%
Python 3 63.55 844,180 1947 129.71 40% 71% 33% 61%
fannkuch-redux
source secs mem gz busy cpu load
Go 17.56 1,524 900 70.00 100% 100% 100% 100%
Python 3 534.40 47,236 950 2,104.05 99% 97% 99% 99%
k-nucleotide
source secs mem gz busy cpu load
Go 11.77 160,184 1607 44.52 94% 98% 94% 92%
Python 3 72.24 199,856 1967 275.38 94% 94% 96% 96%
reverse-complement
source secs mem gz busy cpu load
Go 3.83 1,782,940 1338 6.67 21% 74% 16% 62%
Python 3 16.93 1,777,852 434 17.58 78% 21% 4% 0%
binary-trees
source secs mem gz busy cpu load
Go 25.17 361,152 1005 86.86 87% 89% 86% 84%
Python 3 80.30 448,004 589 286.50 95% 87% 87% 88%
pidigits
source secs mem gz busy cpu load
Go 2.10 8,448 603 2.17 1% 48% 55% 0%
Python 3 3.47 10,356 386 3.53 1% 1% 0% 100%
regex-redux
source secs mem gz busy cpu load
Go 29.82 428,224 802 62.65 48% 45% 52% 65%
Python 3 18.45 457,340 512 37.52 69% 37% 50% 48%
Go go version go1.13 linux/amd64
Python 3 Python 3.8.0

[Nov 14, 2019] perl - package variable scope in module subroutine - Stack Overflow

Nov 14, 2019 | stackoverflow.com

package variable scope in module subroutine Ask Question Asked 7 years, 7 months ago Active 7 years, 7 months ago Viewed 20k times 8 1


brian d foy ,Jul 17, 2014 at 17:54

How do I change the value of a variable in the package used by a module so that subroutines in that module can use it?

Here's my test case:

testmodule.pm:

package testmodule;

use strict;
use warnings;
require Exporter;

our ($VERSION, @ISA, @EXPORT, @EXPORT_OK, %EXPORT_TAGS);

@ISA = qw(Exporter);
@EXPORT = qw(testsub);

my $greeting = "hello testmodule";
my $var2;

sub testsub {
    printf "__PACKAGE__: %s\n", __PACKAGE__;
    printf "\$main::greeting: %s\n", $main::greeting;
    printf "\$greeting: %s\n", $greeting;
    printf "\$testmodule::greeting: %s\n", $testmodule::greeting;
    printf "\$var2: %s\n", $var2;
} # End testsub
1;

testscript.pl:

#!/usr/bin/perl -w
use strict;
use warnings;
use testmodule;

our $greeting = "hello main";
my $var2 = "my var2 in testscript";

$testmodule::greeting = "hello testmodule from testscript";
$testmodule::var2 = "hello var2 from testscript";

testsub();

output:

Name "testmodule::var2" used only once: possible typo at ./testscript.pl line 11.
__PACKAGE__: testmodule
$main::greeting: hello main
$greeting: hello testmodule
$testmodule::greeting: hello testmodule from testscript
Use of uninitialized value $var2 in printf at testmodule.pm line 20.
$var2:

I expected $greeting and $testmodule::greeting to be the same since the package of the subroutine is testmodule .

I guess this has something to do with the way use d modules are eval d as if in a BEGIN block, but I'd like to understand it better.

I was hoping to set the value of the variable from the main script and use it in the module's subroutine without using the fully-qualified name of the variable.

perl-user ,Sep 5, 2013 at 13:58

As you found out, when you use my , you are creating a locally scoped non-package variable. To create a package variable, you use our and not my :
my $foo = "this is a locally scoped, non-package variable";
our $bar = "This is a package variable that's visible in the entire package";

Even better:

{
   my $foo = "This variable is only available in this block";
   our $bar = "This variable is available in the whole package":
}

print "$foo\n";    #Whoops! Undefined variable
print "$bar\n";    #Bar is still defined even out of the block

When you don't put use strict in your program, all variables defined are package variables. That's why when you don't put it, it works the way you think it should and putting it in breaks your program.

However, as you can see in the following example, using our will solve your dilemma:

File Local/Foo.pm
#! /usr/local/bin perl
package Local::Foo;

use strict;
use warnings;
use feature qw(say);

use Exporter 'import';
our @EXPORT = qw(testme);

our $bar = "This is the package's bar value!";
sub testme {

    # $foo is a locally scoped, non-package variable. It's undefined and an error
    say qq(The value of \$main::foo is "$main::foo");

    # $bar is defined in package main::, and will print out
    say qq(The value of \$main::bar is "$main::bar");

    # These both refer to $Local::Foo::bar
    say qq(The value of \$Local::Foo::bar is "$Local::Foo::bar");
    say qq(The value of bar is "$bar");
}

1;
File test.pl
#! /usr/local/bin perl
use strict;
use warnings;
use feature qw(say);
use Local::Foo;

my $foo = "This is foo";
our $bar = "This is bar";
testme;

say "";
$Local::Foo::bar = "This is the NEW value for the package's bar";
testme

And, the output is:

Use of uninitialized value $foo in concatenation (.) or string at Local/Foo.pm line 14.
The value of $main::foo is ""
The value of $main::bar is "This is bar"
The value of $Local::Foo::bar is "This is the package's bar value!"
The value of bar is "This is the package's bar value!"

Use of uninitialized value $foo in concatenation (.) or string at Local/Foo.pm line 14.
The value of $main::foo is ""
The value of $main::bar is "This is bar"
The value of $Local::Foo::bar is "This is the NEW value for the package's bar"
The value of bar is "This is the NEW value for the package's bar"

The error message you're getting is the result of $foo being a local variable, and thus isn't visible inside the package. Meanwhile, $bar is a package variable and is visible.

Sometimes, it can be a bit tricky:

if ($bar -eq "one") {
   my $foo = 1;
}
else {
   my $foo = 2;
}

print "Foo = $foo\n";

That doesn't work because $foo only bas a value inside the if block. You have to do this:

my $foo;
if ($bar -eq "one") {
   $foo = 1;
}
else {
  $foo = 2;
}

print "Foo = $foo\n"; #This works!

Yes, it can be a bit to get your head wrapped around it initially, but the use of use strict; and use warnings; is now de rigueur and for good reasons. The use of use strict; and use warnings; probably has eliminated 90% of the mistakes people make in Perl. You can't make a mistake of setting the value of $foo in one part of the program, and attempting to use $Foo in another. It's one of the things I really miss in Python.

> ,

After reading Variable Scoping in Perl: the basics more carefully, I realized that a variable declared with my isn't in the current package. For example, in a simple script with no modules if I declare my $var = "hello" $main::var still doesn't have a value.

The way that this applies in this case is in the module. Since my $greeting is declared in the file, that hides the package's version of $greeting and that's the value which the subroutine sees. If I don't declare the variable first, the subroutine would see the package variable, but it doesn't get that far because I use strict .

If I don't use strict and don't declare my $greeting , it works as I would have expected. Another way to get the intended value and not break use strict is to use our $greeting . The difference being that my declares a variable in the current scope while our declares a variable in the current package .

[Nov 13, 2019] How fast is Perl s smartmatch operator when searching for a scalar in an array

Nov 13, 2019 | stackoverflow.com

Paul Tomblin ,Oct 19, 2010 at 13:38

I want to repeatedly search for values in an array that does not change.

So far, I have been doing it this way: I put the values in a hash (so I have an array and a hash with essentially the same contents) and I search the hash using exists .

I don't like having two different variables (the array and the hash) that both store the same thing; however, the hash is much faster for searching.

I found out that there is a ~~ (smartmatch) operator in Perl 5.10. How efficient is it when searching for a scalar in an array?

> ,

If you want to search for a single scalar in an array, you can use List::Util 's first subroutine. It stops as soon as it knows the answer. I don't expect this to be faster than a hash lookup if you already have the hash , but when you consider creating the hash and having it in memory, it might be more convenient for you to just search the array you already have.

As for the smarts of the smart-match operator, if you want to see how smart it is, test it. :)

There are at least three cases you want to examine. The worst case is that every element you want to find is at the end. The best case is that every element you want to find is at the beginning. The likely case is that the elements you want to find average out to being in the middle.

Now, before I start this benchmark, I expect that if the smart match can short circuit (and it can; its documented in perlsyn ), that the best case times will stay the same despite the array size, while the other ones get increasingly worse. If it can't short circuit and has to scan the entire array every time, there should be no difference in the times because every case involves the same amount of work.

Here's a benchmark:

#!perl
use 5.12.2;
use strict;
use warnings;

use Benchmark qw(cmpthese);

my @hits = qw(A B C);
my @base = qw(one two three four five six) x ( $ARGV[0] || 1 );

my @at_end       = ( @base, @hits );
my @at_beginning = ( @hits, @base );

my @in_middle = @base;
splice @in_middle, int( @in_middle / 2 ), 0, @hits;

my @random = @base;
foreach my $item ( @hits ) {
    my $index = int rand @random;
    splice @random, $index, 0, $item;
    }

sub count {
    my( $hits, $candidates ) = @_;

    my $count;
    foreach ( @$hits ) { when( $candidates ) { $count++ } }
    $count;
    }

cmpthese(-5, {
    hits_beginning => sub { my $count = count( \@hits, \@at_beginning ) },
    hits_end       => sub { my $count = count( \@hits, \@at_end ) },
    hits_middle    => sub { my $count = count( \@hits, \@in_middle ) },
    hits_random    => sub { my $count = count( \@hits, \@random ) },
    control        => sub { my $count = count( [], [] ) },
  }
);
div class="answercell post-layout--right

,

Here's how the various parts did. Note that this is a logarithmic plot on both axes, so the slopes of the plunging lines aren't as close as they look:

So, it looks like the smart match operator is a bit smart, but that doesn't really help you because you still might have to scan the entire array. You probably don't know ahead of time where you'll find your elements. I expect a hash will perform the same as the best case smart match, even if you have to give up some memory for it.


Okay, so the smart match being smart times two is great, but the real question is "Should I use it?". The alternative is a hash lookup, and it's been bugging me that I haven't considered that case.

As with any benchmark, I start off thinking about what the results might be before I actually test them. I expect that if I already have the hash, looking up a value is going to be lightning fast. That case isn't a problem. I'm more interested in the case where I don't have the hash yet. How quickly can I make the hash and lookup a key? I expect that to perform not so well, but is it still better than the worst case smart match?

Before you see the benchmark, though, remember that there's almost never enough information about which technique you should use just by looking at the numbers. The context of the problem selects the best technique, not the fastest, contextless micro-benchmark. Consider a couple of cases that would select different techniques:

Now, keeping those in mind, I add to my previous program:

my %old_hash = map {$_,1} @in_middle; 

cmpthese(-5, {
    ...,
    new_hash       => sub { 
        my %h = map {$_,1} @in_middle; 
        my $count = 0;
        foreach ( @hits ) { $count++ if exists $h{$_} }
        $count;
        },
    old_hash       => sub { 
        my $count = 0;
        foreach ( @hits ) { $count++ if exists $old_hash{$_} }
        $count;
        },
    control_hash   => sub { 
        my $count = 0;
        foreach ( @hits ) { $count++ }
        $count;
        },
    }
);

Here's the plot. The colors are a bit difficult to distinguish. The lowest line there is the case where you have to create the hash any time you want to search it. That's pretty poor. The highest two (green) lines are the control for the hash (no hash actually there) and the existing hash lookup. This is a log/log plot; those two cases are faster than even the smart match control (which just calls a subroutine).

There are a few other things to note. The lines for the "random" case are a bit different. That's understandable because each benchmark (so, once per array scale run) randomly places the hit elements in the candidate array. Some runs put them a bit earlier and some a bit later, but since I only make the @random array once per run of the entire program, they move around a bit. That means that the bumps in the line aren't significant. If I tried all positions and averaged, I expect that "random" line to be the same as the "middle" line.

Now, looking at these results, I'd say that a smart-match is much faster in its worst case than the hash lookup is in its worst case. That makes sense. To create a hash, I have to visit every element of the array and also make the hash, which is a lot of copying. There's no copying with the smart match.

Here's a further case I won't examine though. When does the hash become better than the smart match? That is, when does the overhead of creating the hash spread out enough over repeated searches that the hash is the better choice?

,

Fast for small numbers of potential matches, but not faster than the hash. Hashes are really the right tool for testing set membership. Since hash access is O(log n) and smartmatch on an array is still O(n) linear scan (albeit short-circuiting, unlike grep), with larger numbers of values in the allowed matches, smartmatch gets relatively worse. Benchmark code (matching against 3 values):
#!perl
use 5.12.0;
use Benchmark qw(cmpthese);

my @hits = qw(one two three);
my @candidates = qw(one two three four five six); # 50% hit rate
my %hash;
@hash{@hits} = ();

sub count_hits_hash {
  my $count = 0;
  for (@_) {
    $count++ if exists $hash{$_};
  }
  $count;
}

sub count_hits_smartmatch {
  my $count = 0;
  for (@_) {
    $count++ when @hits;
  }
  $count;
}

say count_hits_hash(@candidates);
say count_hits_smartmatch(@candidates);

cmpthese(-5, {
    hash => sub { count_hits_hash((@candidates) x 1000) },
    smartmatch => sub { count_hits_smartmatch((@candidates) x 1000) },
  }
);
Benchmark results:
             Rate smartmatch       hash
smartmatch  404/s         --       -65%
hash       1144/s       183%         --

[Nov 13, 2019] python - Running shell command and capturing the output

Nov 22, 2012 | stackoverflow.com

Vartec's answer doesn't read all lines, so I made a version that did:

def run_command(command):
    p = subprocess.Popen(command,
                         stdout=subprocess.PIPE,
                         stderr=subprocess.STDOUT)
    return iter(p.stdout.readline, b'')

Usage is the same as the accepted answer:

command = 'mysqladmin create test -uroot -pmysqladmin12'.split()
for line in run_command(command):
    print(line)
share python - Running shell command and capturing the output - Stack Overflow Share a link to this answer Copy link | improve this answer edited May 23 '17 at 11:33 Community ♦ 1 1 1 silver badge answered Oct 30 '12 at 9:24 Max Ekman Max Ekman 769 5 5 silver badges 5 5 bronze badges

[Nov 13, 2019] Execute shell commands in Python

Nov 13, 2019 | unix.stackexchange.com

Execute shell commands in Python Ask Question Asked 4 years ago Active 2 months ago Viewed 557k times 67 32


fooot ,Nov 8, 2017 at 21:39

I'm currently studying penetration testing and Python programming. I just want to know how I would go about executing a Linux command in Python. The commands I want to execute are:
echo 1 > /proc/sys/net/ipv4/ip_forward
iptables -t nat -A PREROUTING -p tcp --destination-port 80 -j REDIRECT --to-port 8080

If I just use print in Python and run it in the terminal will it do the same as executing it as if you was typing it yourself and pressing Enter ?

binarysubstrate ,Feb 28 at 19:58

You can use os.system() , like this:
import os
os.system('ls')

Or in your case:

os.system('echo 1 > /proc/sys/net/ipv4/ip_forward')
os.system('iptables -t nat -A PREROUTING -p tcp --destination-port 80 -j REDIRECT --to-port 8080')

Better yet, you can use subprocess's call, it is safer, more powerful and likely faster:

from subprocess import call
call('echo "I like potatos"', shell=True)

Or, without invoking shell:

call(['echo', 'I like potatos'])

If you want to capture the output, one way of doing it is like this:

import subprocess
cmd = ['echo', 'I like potatos']
proc = subprocess.Popen(cmd, stdout=subprocess.PIPE, stderr=subprocess.PIPE)

o, e = proc.communicate()

print('Output: ' + o.decode('ascii'))
print('Error: '  + e.decode('ascii'))
print('code: ' + str(proc.returncode))

I highly recommend setting a timeout in communicate , and also to capture the exceptions you can get when calling it. This is a very error-prone code, so you should expect errors to happen and handle them accordingly.

https://docs.python.org/3/library/subprocess.html

jordanm ,Oct 23, 2015 at 15:43

The first command simply writes to a file. You wouldn't execute that as a shell command because python can read and write to files without the help of a shell:
with open('/proc/sys/net/ipv4/ip_forward', 'w') as f:
    f.write("1")

The iptables command is something you may want to execute externally. The best way to do this is to use the subprocess module .

import subprocess
subprocess.check_call(['iptables', '-t', 'nat', '-A',
                       'PREROUTING', '-p', 'tcp', 
                       '--destination-port', '80',
                       '-j', 'REDIRECT', '--to-port', '8080'])

Note that this method also does not use a shell, which is unnecessary overhead.

Tom Hunt ,Oct 23, 2015 at 15:41

The quickest way:
import os
os.system("your command here")

This isn't the most flexible approach; if you need any more control over your process than "run it once, to completion, and block until it exits", then you should use the subprocess module instead.

jordanm ,Apr 5, 2018 at 9:23

As a general rule, you'd better use python bindings whenever possible (better Exception catching, among other advantages.)

For the echo command, it's obviously better to use python to write in the file as suggested in @jordanm's answer.

For the iptables command, maybe python-iptables ( PyPi page , GitHub page with description and doc ) would provide what you need (I didn't check your specific command).

This would make you depend on an external lib, so you have to weight the benefits. Using subprocess works, but if you want to use the output, you'll have to parse it yourself, and deal with output changes in future iptables versions.

,

A python version of your shell. Be careful, I haven't tested it.
from subprocess import run

def bash(command):
        run(command.split())

>>> bash('find / -name null')
/dev/null
/sys/fs/selinux/null
/sys/devices/virtual/mem/null
/sys/class/mem/null
/usr/lib/kbd/consoletrans/null

[Nov 13, 2019] Static code analysis module in Perl - Stack Overflow

Nov 13, 2019 | stackoverflow.com

Static code analysis module in Perl Ask Question Asked 7 years, 5 months ago Active 1 year, 7 months ago Viewed 835 times 0

DavidO ,Jun 12, 2012 at 9:13

Is there any static code analysis module in Perl except B::Lint and Perl::Critic? How effective is Module::Checkstyle?

> ,

There is a post on perlmonks.org asking if PPI can be used for static analysis. PPI is the power behind Perl::Critic, according to the reviews of this module. (I have not used it yet).

Then there is perltidy .

[Nov 12, 2019] c - python (conditional-ternary) operator for assignments

Jul 30, 2014 | stackoverflow.com
This question already has an answer here:

karadoc ,May 14, 2013 at 13:01

Python has such an operator:
variable = something if condition else something_else

Alternatively, although not recommended (see @karadoc's comment):

variable = (condition and something) or something_else

[Nov 12, 2019] Static code analysis module in Perl - Stack Overflow

Nov 12, 2019 | stackoverflow.com

Static code analysis module in Perl Ask Question Asked 7 years, 5 months ago Active 1 year, 7 months ago Viewed 835 times 0

DavidO ,Jun 12, 2012 at 9:13

Is there any static code analysis module in Perl except B::Lint and Perl::Critic? How effective is Module::Checkstyle?

> ,

There is a post on perlmonks.org asking if PPI can be used for static analysis. PPI is the power behind Perl::Critic, according to the reviews of this module. (I have not used it yet).

Then there is perltidy .

[Nov 11, 2019] How fast is Perl's smartmatch operator when searching for a scalar in an array - Stack Overflow

Nov 11, 2019 | stackoverflow.com

[Nov 11, 2019] python - How to access environment variable values

Nov 11, 2019 | stackoverflow.com
Environment variables are accessed through os.environ
import os
print(os.environ['HOME'])

Or you can see a list of all the environment variables using:

os.environ

As sometimes you might need to see a complete list!

# using get will return `None` if a key is not present rather than raise a `KeyError`
print(os.environ.get('KEY_THAT_MIGHT_EXIST'))

# os.getenv is equivalent, and can also give a default value instead of `None`
print(os.getenv('KEY_THAT_MIGHT_EXIST', default_value))

Python default installation on Windows is C:\Python . If you want to find out while running python you can do:

import sys
print(sys.prefix)

,

import sys
print sys.argv[0]

This will print foo.py for python foo.py , dir/foo.py for python dir/foo.py , etc. It's the first argument to python . (Note that after py2exe it would be foo.exe .)

[Nov 11, 2019] How can I find the current OS in Python

Nov 11, 2019 | stackoverflow.com

> ,May 29, 2012 at 21:57

Possible Duplicate:
Python: What OS am I running on?

As the title says, how can I find the current operating system in python?

Shital Shah ,Sep 23 at 23:34

I usually use sys.platform ( docs ) to get the platform. sys.platform will distinguish between linux, other unixes, and OS X, while os.name is " posix " for all of them.

For much more detailed information, use the platform module . This has cross-platform functions that will give you information on the machine architecture, OS and OS version, version of Python, etc. Also it has os-specific functions to get things like the particular linux distribution.

xssChauhan ,Sep 9 at 7:34

If you want user readable data but still detailed, you can use platform.platform()
>>> import platform
>>> platform.platform()
'Linux-3.3.0-8.fc16.x86_64-x86_64-with-fedora-16-Verne'

platform also has some other useful methods:

>>> platform.system()
'Windows'
>>> platform.release()
'XP'
>>> platform.version()
'5.1.2600'

Here's a few different possible calls you can make to identify where you are

import platform
import sys

def linux_distribution():
  try:
    return platform.linux_distribution()
  except:
    return "N/A"

print("""Python version: %s
dist: %s
linux_distribution: %s
system: %s
machine: %s
platform: %s
uname: %s
version: %s
mac_ver: %s
""" % (
sys.version.split('\n'),
str(platform.dist()),
linux_distribution(),
platform.system(),
platform.machine(),
platform.platform(),
platform.uname(),
platform.version(),
platform.mac_ver(),))

The outputs of this script ran on a few different systems (Linux, Windows, Solaris, MacOS) and architectures (x86, x64, Itanium, power pc, sparc) is available here: https://github.com/hpcugent/easybuild/wiki/OS_flavor_name_version

Steg ,Mar 31, 2015 at 15:13

import os
print os.name

This gives you the essential information you will usually need. To distinguish between, say, different editions of Windows, you will have to use a platform-specific method.

UnkwnTech ,Sep 21, 2008 at 6:17

https://docs.python.org/library/os.html

To complement Greg's post, if you're on a posix system, which includes MacOS, Linux, Unix, etc. you can use os.uname() to get a better feel for what kind of system it is.

> ,

Something along the lines:
import os
if (os.name == "posix"):
    print os.system("uname -a")
# insert other possible OSes here
# ...
else:
    print "unknown OS"

[Nov 11, 2019] Python read file from current line

Nov 11, 2019 | stackoverflow.com

Ask Question Asked 6 years, 8 months ago Active 6 years, 8 months ago Viewed 2k times 0


dylanoo ,Feb 18, 2013 at 17:09

I have one problem regarding using python to process the trace file (it contains billion lines of data).

What I want to do is, the program will find one specific line in the file (say it is line# x), and it needs to find another symbol from this (line# x) in the file. Once it finds the line, starts from (line# x) again to search another one.

What I did now, is as following, but the problem is it always needs to reopen the file and read from the beginning to find the match ones (line # > x, and contain the symbol I want). For one big trace file, it takes too long to processing.

1.

    for line in file.readlines()
      i++ #update the line number
      if i > x:
          if (line.find()):

or:

   for i, line in enumerate(open(file)):
      if i > x:
          if ....

Anyone can give me one hint on better ideas?

Thanks

dylanoo ,Feb 18, 2013 at 20:24

If the file is otherwise stable, use fileobj.tell() to remember your position in the file, then next time use fileobj.seek(pos) to return to that same position in the file.

This only works if you do not use the fileobject as an iterator (no for line in fileobject) or next(fileobject) ) as that uses a read-ahead buffer that will obscure the exact position.

Instead, use:

for line in iter(fileobj.readline, ''):

to still use fileobj in an iteration context.

Martijn Pieters ♦ ,Feb 18, 2013 at 17:30

I suggest you use random access, and record where your line started. Something like:
index = []

fh = open(gash.txt)

for line in fh:
    if target in line:
        index.append(fh.tell() - len(line))

Then, when you want to recall the contents, use fh.seek(index[n]) .

A couple of "gotchas":

  1. Notice that the index position will not be the same as the line number. If you need the line number then maybe use a dictionary, with the line number as the key.
  2. On Windows, you will have to adjust the file position by -1. This is because the "\r" is stripped out and does not appear in the len(line) .

[Nov 10, 2019] Perl to Python Function translation [closed]

Feb 01, 2014 | stackoverflow.com

Ask Question Asked 5 years, 8 months ago Active 5 years, 8 months ago Viewed 303 times -3


Jim Garrison ,Feb 1, 2014 at 22:24

I am trying to translate a Perl function into a Python function, but I am having trouble figuring out what some of the Perl to Python function equivalents.

Perl function:

sub reverse_hex {

 my $HEXDATE = shift;
 my @bytearry=();
 my $byte_cnt = 0;
 my $max_byte_cnt = 8;
 my $byte_offset = 0;
 while($byte_cnt < $max_byte_cnt) {
   my $tmp_str = substr($HEXDATE,$byte_offset,2);
    push(@bytearry,$tmp_str);
   $byte_cnt++;
   $byte_offset+=2;
 }
   return join('',reverse(@bytearry));
}

I am not sure what "push", "shift", and "substr" are doing here that would be the same in Python.

Any help will be much appreciated.

Kenosis ,Feb 1, 2014 at 22:17

The Perl subroutine seems rather complicated for what it does, viz., taking chunks of two chars at a time (the first 16 chars) from the sent string and then reverses it. Another Perl option is:
sub reverse_hex {
    return join '', reverse unpack 'A2' x 8, $_[0];
}

First, unpack here takes two characters at a time (eight times) and produces a list. That list is reverse d and join ed to produce the final string.

Here's a Python subroutine to accomplish this:

def reverse_hex(HEXDATE):
    hexVals = [HEXDATE[i:i + 2] for i in xrange(0, 16, 2)]
    reversedHexVals = hexVals[::-1]
    return ''.join(reversedHexVals)

The list comprehension produces eight elements of two characters each. [::-1] reverses the list's elements and the result is join ed and returned.

Hope this helps!

MikeMayer67 ,Feb 2, 2014 at 2:10

I realize that you are asking about the perl to python translation, but if you have any control over the perl, I would like to point out that this function is a lot more complicated than it needs to be.

The entire thing could be replaced with:

sub reverse_hex
{
  my $hexdate = shift;
  my @bytes = $hexdate =~ /../g;  # break $hexdate into array of character pairs
  return join '', reverse(@bytes);
}

Not only is this shorter, it is much easier to get your head around. Of course, if you have no control over the perl, you are stuck with what you were dealt.

[Nov 08, 2019] Is losing BDFL a death sentence for open source projects such as Python? by Jason Baker

Jul 16, 2018 | opensource.com
What happens when a Benevolent Dictator For Life moves on from an open source project? up 2 comments Image credits : Original photo by Gabriel Kamener, Sown Together, Modified by Jen Wike Huger x Subscribe now

Get the highlights in your inbox every week.

https://opensource.com/eloqua-embedded-email-capture-block.html?offer_id=70160000000QzXNAA0 Guido van Rossum , creator of the Python programming language and Benevolent Dictator For Life (BDFL) of the project, announced his intention to step away.

Below is a portion of his message, although the entire email is not terribly long and worth taking the time to read if you're interested in the circumstances leading to van Rossum's departure.

I would like to remove myself entirely from the decision process. I'll still be there for a while as an ordinary core dev, and I'll still be available to mentor people -- possibly more available. But I'm basically giving myself a permanent vacation from being BDFL, and you all will be on your own.

After all that's eventually going to happen regardless -- there's still that bus lurking around the corner, and I'm not getting younger... (I'll spare you the list of medical issues.)

I am not going to appoint a successor.

So what are you all going to do? Create a democracy? Anarchy? A dictatorship? A federation?

It's worth zooming out for a moment to consider the issue at a larger scale. How an open source project is governed can have very real consequences on the long-term sustainability of its user and developer communities alike.

BDFLs tend to emerge from passion projects, where a single individual takes on a project before growing a community around it. Projects emerging from companies or other large organization often lack this role, as the distribution of authority is more formalized, or at least more dispersed, from the start. Even then, it's not uncommon to need to figure out how to transition from one form of project governance to another as the community grows and expands.

More Python Resources

But regardless of how an open source project is structured, ultimately, there needs to be some mechanism for deciding how to make technical decisions. Someone, or some group, has to decide which commits to accept, which to reject, and more broadly what direction the project is going to take from a technical perspective.

Surely the Python project will be okay without van Rossum. The Python Software Foundation has plenty of formalized structure in place bringing in broad representation from across the community. There's even been a humorous April Fools Python Enhancement Proposal (PEP) addressing the BDFL's retirement in the past.

That said, it's interesting that van Rossum did not heed the fifth lesson of Eric S. Raymond from his essay, The Mail Must Get Through (part of The Cathedral & the Bazaar ) , which stipulates: "When you lose interest in a program, your last duty to it is to hand it off to a competent successor." One could certainly argue that letting the community pick its own leadership, though, is an equally-valid choice.

What do you think? Are projects better or worse for being run by a BDFL? What can we expect when a BDFL moves on? And can someone truly step away from their passion project after decades of leading it? Will we still turn to them for the hard decisions, or can a community smoothly transition to new leadership without the pitfalls of forks or lost participants?

Can you truly stop being a BDFL? Or is it a title you'll hold, at least informally, until your death? Topics Community management Python 2018 Open Source Yearbook Yearbook About the author Jason Baker - I use technology to make the world more open. Linux desktop enthusiast. Map/geospatial nerd. Raspberry Pi tinkerer. Data analysis and visualization geek. Occasional coder. Cloud nativist. Civic tech and open government booster. More about me

Recommended reading
Conquering documentation challenges on a massive project

4 Python tools for getting started with astronomy

5 reasons why I love Python

Building trust in the Linux community

Pylint: Making your Python code consistent

Perceiving Python programming paradigms
2 Comments

Mike James on 17 Jul 2018 Permalink

My take on the issue:
https://www.i-programmer.info/news/216-python/11967-guido-van-rossum-qui...

Maxim Stewart on 05 Aug 2018 Permalink

"So what are you all going to do? Create a democracy? Anarchy? A dictatorship? A federation?"

Power coalesced to one point is always scary when thought about in the context of succession. A vacuum invites anarchy and I often think about this for when Linus Torvalds leaves the picture. We really have no concrete answers for what is the best way forward but my hope is towards a democratic process. But, as current history indicates, a democracy untended by its citizens invites quite the nightmare and so too does this translate to the keeping up of a project.

[Nov 08, 2019] How to escape unicode characters in bash prompt correctly - Stack Overflow

Nov 08, 2019 | stackoverflow.com

How to escape unicode characters in bash prompt correctly Ask Question Asked 8 years, 2 months ago Active 9 months ago Viewed 6k times 7 2


Andy Ray ,Aug 18, 2011 at 19:08

I have a specific method for my bash prompt, let's say it looks like this:
CHAR="༇ "
my_function="
    prompt=\" \[\$CHAR\]\"
    echo -e \$prompt"

PS1="\$(${my_function}) \$ "

To explain the above, I'm builidng my bash prompt by executing a function stored in a string, which was a decision made as the result of this question . Let's pretend like it works fine, because it does, except when unicode characters get involved

I am trying to find the proper way to escape a unicode character, because right now it messes with the bash line length. An easy way to test if it's broken is to type a long command, execute it, press CTRL-R and type to find it, and then pressing CTRL-A CTRL-E to jump to the beginning / end of the line. If the text gets garbled then it's not working.

I have tried several things to properly escape the unicode character in the function string, but nothing seems to be working.

Special characters like this work:

COLOR_BLUE=$(tput sgr0 && tput setaf 6)

my_function="
    prompt="\\[\$COLOR_BLUE\\] \"
    echo -e \$prompt"

Which is the main reason I made the prompt a function string. That escape sequence does NOT mess with the line length, it's just the unicode character.

Andy Ray ,Aug 23, 2011 at 2:09

The \[...\] sequence says to ignore this part of the string completely, which is useful when your prompt contains a zero-length sequence, such as a control sequence which changes the text color or the title bar, say. But in this case, you are printing a character, so the length of it is not zero. Perhaps you could work around this by, say, using a no-op escape sequence to fool Bash into calculating the correct line length, but it sounds like that way lies madness.

The correct solution would be for the line length calculations in Bash to correctly grok UTF-8 (or whichever Unicode encoding it is that you are using). Uhm, have you tried without the \[...\] sequence?

Edit: The following implements the solution I propose in the comments below. The cursor position is saved, then two spaces are printed, outside of \[...\] , then the cursor position is restored, and the Unicode character is printed on top of the two spaces. This assumes a fixed font width, with double width for the Unicode character.

PS1='\['"`tput sc`"'\]  \['"`tput rc`"'༇ \] \$ '

At least in the OSX Terminal, Bash 3.2.17(1)-release, this passes cursory [sic] testing.

In the interest of transparency and legibility, I have ignored the requirement to have the prompt's functionality inside a function, and the color coding; this just changes the prompt to the character, space, dollar prompt, space. Adapt to suit your somewhat more complex needs.

tripleee ,Aug 23, 2011 at 7:01

@tripleee wins it, posting the final solution here because it's a pain to post code in comments:
CHAR="༇"
my_function="
    prompt=\" \\[`tput sc`\\]  \\[`tput rc`\\]\\[\$CHAR\\] \"
    echo -e \$prompt"

PS1="\$(${my_function}) \$ "

The trick as pointed out in @tripleee's link is the use of the commands tput sc and tput rc which save and then restore the cursor position. The code is effectively saving the cursor position, printing two spaces for width, restoring the cursor position to before the spaces, then printing the special character so that the width of the line is from the two spaces, not the character.

> ,

(Not the answer to your problem, but some pointers and general experience related to your issue.)

I see the behaviour you describe about cmd-line editing (Ctrl-R, ... Cntrl-A Ctrl-E ...) all the time, even without unicode chars.

At one work-site, I spent the time to figure out the diff between the terminals interpretation of the TERM setting VS the TERM definition used by the OS (well, stty I suppose).

NOW, when I have this problem, I escape out of my current attempt to edit the line, bring the line up again, and then immediately go to the 'vi' mode, which opens the vi editor. (press just the 'v' char, right?). All the ease of use of a full-fledged session of vi; why go with less ;-)?

Looking again at your problem description, when you say

my_function="
    prompt=\" \[\$CHAR\]\"
    echo -e \$prompt"

That is just a string definition, right? and I'm assuming your simplifying the problem definition by assuming this is the output of your my_function . It seems very likely in the steps of creating the function definition, calling the function AND using the values returned are a lot of opportunities for shell-quoting to not work the way you want it to.

If you edit your question to include the my_function definition, and its complete use (reducing your function to just what is causing the problem), it may be easier for others to help with this too. Finally, do you use set -vx regularly? It can help show how/wnen/what of variable expansions, you may find something there.

Failing all of those, look at Orielly termcap & terminfo . You may need to look at the man page for your local systems stty and related cmds AND you may do well to look for user groups specific to you Linux system (I'm assuming you use a Linux variant).

I hope this helps.

[Nov 07, 2019] Is BDFL a death sentence Opensource.com

Nov 07, 2019 | opensource.com

What happens when a Benevolent Dictator For Life moves on from an open source project? 16 Jul 2018 Jason Baker (Red Hat) Feed 131 up 2 comments Image credits : Original photo by Gabriel Kamener, Sown Together, Modified by Jen Wike Huger x Subscribe now

Get the highlights in your inbox every week.

https://opensource.com/eloqua-embedded-email-capture-block.html?offer_id=70160000000QzXNAA0 Guido van Rossum , creator of the Python programming language and Benevolent Dictator For Life (BDFL) of the project, announced his intention to step away.

Below is a portion of his message, although the entire email is not terribly long and worth taking the time to read if you're interested in the circumstances leading to van Rossum's departure.

I would like to remove myself entirely from the decision process. I'll still be there for a while as an ordinary core dev, and I'll still be available to mentor people -- possibly more available. But I'm basically giving myself a permanent vacation from being BDFL, and you all will be on your own.

After all that's eventually going to happen regardless -- there's still that bus lurking around the corner, and I'm not getting younger... (I'll spare you the list of medical issues.)

I am not going to appoint a successor.

So what are you all going to do? Create a democracy? Anarchy? A dictatorship? A federation?

It's worth zooming out for a moment to consider the issue at a larger scale. How an open source project is governed can have very real consequences on the long-term sustainability of its user and developer communities alike.

BDFLs tend to emerge from passion projects, where a single individual takes on a project before growing a community around it. Projects emerging from companies or other large organization often lack this role, as the distribution of authority is more formalized, or at least more dispersed, from the start. Even then, it's not uncommon to need to figure out how to transition from one form of project governance to another as the community grows and expands.

More Python Resources

But regardless of how an open source project is structured, ultimately, there needs to be some mechanism for deciding how to make technical decisions. Someone, or some group, has to decide which commits to accept, which to reject, and more broadly what direction the project is going to take from a technical perspective.

Surely the Python project will be okay without van Rossum. The Python Software Foundation has plenty of formalized structure in place bringing in broad representation from across the community. There's even been a humorous April Fools Python Enhancement Proposal (PEP) addressing the BDFL's retirement in the past.

That said, it's interesting that van Rossum did not heed the fifth lesson of Eric S. Raymond from his essay, The Mail Must Get Through (part of The Cathedral & the Bazaar ) , which stipulates: "When you lose interest in a program, your last duty to it is to hand it off to a competent successor." One could certainly argue that letting the community pick its own leadership, though, is an equally-valid choice.

What do you think? Are projects better or worse for being run by a BDFL? What can we expect when a BDFL moves on? And can someone truly step away from their passion project after decades of leading it? Will we still turn to them for the hard decisions, or can a community smoothly transition to new leadership without the pitfalls of forks or lost participants?

Can you truly stop being a BDFL? Or is it a title you'll hold, at least informally, until your death? Topics Community management Python 2018 Open Source Yearbook Yearbook About the author Jason Baker - I use technology to make the world more open. Linux desktop enthusiast. Map/geospatial nerd. Raspberry Pi tinkerer. Data analysis and visualization geek. Occasional coder. Cloud nativist. Civic tech and open government booster. More about me

Recommended reading
Conquering documentation challenges on a massive project

4 Python tools for getting started with astronomy

5 reasons why I love Python

Building trust in the Linux community

Pylint: Making your Python code consistent

Perceiving Python programming paradigms
2 Comments

Mike James on 17 Jul 2018 Permalink

My take on the issue:
https://www.i-programmer.info/news/216-python/11967-guido-van-rossum-qui...

Maxim Stewart on 05 Aug 2018 Permalink

"So what are you all going to do? Create a democracy? Anarchy? A dictatorship? A federation?"

Power coalesced to one point is always scary when thought about in the context of succession. A vacuum invites anarchy and I often think about this for when Linus Torvalds leaves the picture. We really have no concrete answers for what is the best way forward but my hope is towards a democratic process. But, as current history indicates, a democracy untended by its citizens invites quite the nightmare and so too does this translate to the keeping up of a project.

[Nov 02, 2019] Copied variable changes the original

Nov 14, 2011 | stackoverflow.com

Copied variable changes the original? Ask Question Asked 7 years, 11 months ago Active 2 years, 9 months ago Viewed 61k times 46 17


André Freitas ,Nov 14, 2011 at 13:56

I have a simple problem in Python that is very very strange.
def estExt(matriz,erro):
    # (1) Determinar o vector X das soluções
    print ("Matrix after:");
    print(matriz);

    aux=matriz;
    x=solucoes(aux); # IF aux is a copy of matrix, why the matrix is changed??

    print ("Matrix before: ");
    print(matriz)

...

As you see below, the matrix matriz is changed in spite of the fact that aux is the one being changed by the function solucoes() .

Matrix before:
[[7, 8, 9, 24], [8, 9, 10, 27], [9, 10, 8, 27]]

Matrix after:
[[7, 8, 9, 24], [0.0, -0.14285714285714235, -0.2857142857142847, -0.42857142857142705], [0.0, 0.0, -3.0, -3.0000000000000018]]

André Freitas ,Nov 14, 2011 at 17:16

The line
aux=matriz;

Does not make a copy of matriz , it merely creates a new reference to matriz named aux . You probably want

aux=matriz[:]

Which will make a copy, assuming matriz is a simple data structure. If it is more complex, you should probably use copy.deepcopy

aux = copy.deepcopy(matriz)

As an aside, you don't need semi-colons after each statement, python doesn't use them as EOL markers.

André Freitas ,Nov 15, 2011 at 8:49

Use copy module
aux = copy.deepcopy(matriz) # there is copy.copy too for shallow copying

Minor one: semicolons are not needed.

aux is not a copy of matrix , it's just a different name that refers to the same object.

[Oct 25, 2019] unix - Remove a file on Linux using the inode number - Super User

Oct 25, 2019 | superuser.com

,

ome other methods include:

escaping the special chars:

[~]$rm \"la\*

use the find command and only search the current directory. The find command can search for inode numbers, and has a handy -delete switch:

[~]$ls -i
7404301 "la*

[~]$find . -maxdepth 1 -type f -inum 7404301
./"la*

[~]$find . -maxdepth 1 -type f -inum 7404301 -delete
[~]$ls -i
[~]$

,

Maybe I'm missing something, but...
rm '"la*'

Anyways, filenames don't have inodes, files do. Trying to remove a file without removing all filenames that point to it will damage your filesystem.

[Oct 22, 2019] Perl vs Python log processing performance

Oct 22, 2019 | stackoverflow.com

Ask Question Asked 6 years, 11 months ago Active 6 years, 11 months ago Viewed 2k times 0 0

texasbruce ,Nov 11, 2012 at 2:05

I am working on a web-based log management system that will be built on the Grails framework and I am going to use one of the text processing languages like Python or Perl. I have created Python and Perl scripts that load log files and parse each line to save them to a MySQL database (the file contains about 40,000 lines, about 7MB). It took 1 min 2 secs using Perl and only 17 secs using Python .

I had supposed that Perl would be faster than Python, as Perl is the original text processing language (my suspicions also coming from different blogs where I was reading about Perl text processing performance).

Also I was not expecting a 47 second difference between Perl and Python. Why is Perl taking more time than Python to process my log file? Is it because I am using some wrong db module or my code and regular expression for Perl can be improved?

Note: I am a Java and Groovy developer and I have no experience with Perl (I am using Strawberry Perl v5.16). Also I have made this test with Java (1 min 5 secs) and Groovy (1 min 7 secs) but more than 1 min to process the log file is too much, so both languages are out and now I want to choose between Perl and Python.

PERL Code

use DBI;
use DBD::mysql;
# make connection to database
$connection = DBI->connect("dbi:mysql:logs:localhost:3306","root","") || die      "Cannot connect: $DBI::errstr";

# set the value of your SQL query
$query = "insert into logs (line_number, dated, time_stamp, thread, level, logger, user, message)
        values (?, ?, ?, ?, ?, ?, ?, ?) ";

# prepare your statement for connecting to the database
$statement = $connection->prepare($query); 

$runningTime = time;

# open text file
open (LOG,'catalina2.txt') || die "Cannot read logfile!\n";;

while (<LOG>) {
    my ($date, $time, $thread, $level, $logger, $user, $message) = /^(\d{4}-\d{2}-\d{2}) (\d{2}:\d{2}:\d{2},\d{3}) (\[.*\]) (.*) (\S*) (\(.*\)) - (.*)$/;

    $statement->execute(1, $date, $time, $thread, $level, $logger, $user, $message);
}  

# close the open text file
close(LOG);

# close database connection
$connection->disconnect;

$runningTime = time - $runningTime;
printf("\n\nTotal running time: %02d:%02d:%02d\n\n", int($runningTime / 3600),   int(($runningTime % 3600) / 60), int($runningTime % 60));

# exit the script
exit;

PYTHON Code

import re
import mysql.connector
import time

file = open("D:\catalina2.txt","r")
rexp = re.compile('^(\d{4}-\d{2}-\d{2}) (\d{2}:\d{2}:\d{2},\d{3}) (\[.*\]) (.*) (\S*) (\(.*\)) - (.*)$')
conn = mysql.connector.connect(user='root',host='localhost',database='logs')
cursor = conn.cursor()

tic = time.clock()

increment  = 1
for text in file.readlines():
    match = rexp.match(text)
    increment +=  1
cursor.execute('insert into logs (line_number,dated, time_stamp, thread,level,logger,user,message ) values (%s,%s,%s,%s,%s,%s,%s,%s)', (increment, match.group(1), match.group(2),match.group(3),match.group(4),match.group(5),match.group(6),match.group(7)))

conn.commit()
cursor.close()
conn.close()

toc = time.clock()
print "Total time: %s" % (toc - tic)

David-SkyMesh ,Nov 11, 2012 at 1:35

It is not a fair comparison:

You are only calling cursor.execute once in Python:

for text in file.readlines():
    match = rexp.match(text)
    increment +=  1
cursor.execute('insert into logs (line_number,dated, time_stamp, thread,level,logger,user,message ) values (%s,%s,%s,%s,%s,%s,%s,%s)', (increment, match.group(1), match.group(2),match.group(3),match.group(4),match.group(5),match.group(6),match.group(7)))

But you are calling $statement->execute many times in Perl:

while (<LOG>) {
    my ($date, $time, $thread, $level, $logger, $user, $message) = /^(\d{4}-\d{2}-\d{2}) (\d{2}:\d{2}:\d{2},\d{3}) (\[.*\]) (.*) (\S*) (\(.*\)) - (.*)$/;

    $statement->execute(1, $date, $time, $thread, $level, $logger, $user, $message);
}

By the way, for the Python version, calling cursor.execute once for every row will be slow. You can make it faster by using cursor.executemany :

sql = 'insert into logs (line_number,dated, time_stamp, thread,level,logger,user,message ) values (%s,%s,%s,%s,%s,%s,%s,%s)'
args = []
for text in file:
    match = rexp.match(text)
    increment +=  1
    args.append([increment] + list(match.groups()))

cursor.executemany(sql, args)

If there are too many lines in the log file, you may need to break this up into blocks:

args = []
for text in file:
    match = rexp.match(text)
    increment +=  1
    args.append([increment] + list(match.groups()))
    if increment % 1000 == 0:
        cursor.executemany(sql, args)
        args = []
if args:
    cursor.executemany(sql, args)

(Also, don't use file.readlines() because this creates a list (which may be huge). file is an iterator which spits out one line at a time, so for text in file suffices.)

[Oct 22, 2019] Is there an advantage to using Bash over Perl or Python?

Oct 22, 2019 | stackoverflow.com

Ask Question Asked 8 years, 5 months ago Active 8 years, 5 months ago Viewed 19k times 23 10


> ,May 2, 2011 at 18:58

Hey I've been using Linux for a while and thought it was time to finally dive into shell scripting.

The problem is I've failed to find any significant advantage of using Bash over something like Perl or Python. Are there any performance or power differences between the two? I'd figure Python/Perl would be more well suited as far as power and efficiency goes.

Sebastian ,May 2, 2011 at 15:21

Two advantages come to mind:

By the way, I usually have some python calls in my bash scripts (e.g. for plotting). Use whatever is best for the task!

Mario Peshev ,May 2, 2011 at 15:16

Perl scripts are usually (if not 100% of the times) faster than bash.

A discussion on that: Perl vs Bash

reinierpost ,May 7, 2011 at 12:16

bash isn't a language so much as a command interpreter that's been hacked to death to allow for things that make it look like a scripting language. It's great for the simplest 1-5 line one-off tasks, but things that are dead simple in Perl or Python like array manipulation are horribly ugly in bash. I also find that bash tends not to pass two critical rules of thumb:
  1. The 6-month rule, which says you should be able to easily discern the purpose and basic mechanics of a script you wrote but haven't looked at in 6 months.
  2. The 'WTF per minute' rule. Everyone has their limit, and mine is pretty small. Once I get to 3 WTFs/min, I'm looking elsewhere.

As for 'shelling out' in scripting languages like Perl and Python, I find that I almost never need to do this, fwiw (disclaimer: I code almost 100% in Python). The Python os and shutil modules have most of what I need most of the time, and there are built-in modules for handling tarfiles, gzip files, zip files, etc. There's a glob module, an fnmatch module... there's a lot of stuff there. If you come across something you need to parallelize, then indent your code a level, put it in a 'run()' method, put that in a class that extends either threading.Thread or multiprocessing.Process, instantiate as many of those as you want, calling 'start()' on each one. Less than 5 minutes to get parallel execution generally.

Best of luck. Hope this helps.

daotoad ,May 2, 2011 at 17:40

For big projects use a language like Perl.

There are a few things you can only do in bash (for example, alter the calling environment (when a script is sourced rather than run). Also, shell scripting is commonplace. It is worthwhile to learn the basics and learn your way around the available docs.

Plus there are times when knowing a shell well can save your bacon (on a fork-bombed system where you can't start any new processes, or if /usr/bin and or /usr/local/bin fail to mount).

Sebastian ,May 3, 2011 at 8:47

The advantage is that it's right there. Unless you use Python (or Perl) as your shell, writing a script to do a simple loop is a bunch of extra work.

For short, simple scripts that call other programs, I'll use Bash. If I want to keep the output, odds are good that I'll trade up to Python.

For example:

for file in *; do process $file ; done

where process is a program I want to run on each file, or...

while true; do program_with_a_tendency_to_fail ; done

Doing either of those in Python or Perl is overkill.

For actually writing a program that I expect to maintain and use over time, Bash is rarely the right tool for the job. Particularly since most modern Unices come with both Perl and Python.

tchrist ,May 4, 2011 at 11:01

The most important advantage of POSIX shell scripts over Python or Perl scripts is that a POSIX shell is available on virtually every Unix machine. (There are also a few tasks shell scripts happen to be slightly more convenient for, but that's not a major issue.) If the portability is not an issue for you, I don't see much need to learn shell scripting.

tchrist ,May 3, 2011 at 23:50

If you want to execute programs installed on the machine, nothing beats bash. You can always make a system call from Perl or Python, but I find it to be a hassle to read return values, etc.

And since you know it will work pretty much anywhere throughout all of of time...

Alexandr Ciornii ,May 3, 2011 at 8:26

The advantage of shell scripting is that it's globally present on *ix boxes, and has a relatively stable core set of features you can rely on to run everywhere. With Perl and Python you have to worry about whether they're available and if so what version, as there have been significant syntactical incompatibilities throughout their lifespans. (Especially if you include Python 3 and Perl 6.)

The disadvantage of shell scripting is everything else. Shell scripting languages are typically lacking in expressiveness, functionality and performance. And hacking command lines together from strings in a language without strong string processing features and libraries, to ensure the escaping is correct, invites security problems. Unless there's a compelling compatibility reason you need to go with shell, I would personally plump for a scripting language every time.

[Oct 22, 2019] Python Code Glitch May Have Caused Errors In Over 100 Published Studies

Oct 22, 2019 | science.slashdot.org

(vice.com) 121

An anonymous reader quotes Motherboard: The glitch caused results of a common chemistry computation to vary depending on the operating system used, causing discrepancies among Mac, Windows, and Linux systems. The researchers published the revelation and a debugged version of the script, which amounts to roughly 1,000 lines of code, on Tuesday in the journal Organic Letters .

"This simple glitch in the original script calls into question the conclusions of a significant number of papers on a wide range of topics in a way that cannot be easily resolved from published information because the operating system is rarely mentioned," the new paper reads. "Authors who used these scripts should certainly double-check their results and any relevant conclusions using the modified scripts in the [supplementary information]."

Yuheng Luo, a graduate student at the University of Hawaii at Manoa, discovered the glitch this summer when he was verifying the results of research conducted by chemistry professor Philip Williams on cyanobacteria... Under supervision of University of Hawaii at Manoa assistant chemistry professor Rui Sun, Luo used a script written in Python that was published as part of a 2014 paper by Patrick Willoughby, Matthew Jansma, and Thomas Hoye in the journal Nature Protocols . The code computes chemical shift values for NMR, or nuclear magnetic resonance spectroscopy, a common technique used by chemists to determine the molecular make-up of a sample. Luo's results did not match up with the NMR values that Williams' group had previously calculated, and according to Sun, when his students ran the code on their computers, they realized that different operating systems were producing different results.

Sun then adjusted the code to fix the glitch, which had to do with how different operating systems sort files.
The researcher who wrote the flawed script told Motherboard that the new study was "a beautiful example of science working to advance the work we reported in 2014. They did a tremendous service to the community in figuring this out."

Sun described the original authors as "very gracious," saying they encouraged the publication of the findings.

[Oct 22, 2019] Difference in regex behavior between Perl and Python?

Oct 22, 2019 | stackoverflow.com

Ask Question Asked 10 years, 6 months ago Active 10 years, 6 months ago Viewed 2k times 3 1


Gumbo ,Apr 16, 2009 at 18:42

I have a couple email addresses, 'support@company.com' and '1234567@tickets.company.com' .

In perl, I could take the To: line of a raw email and find either of the above addresses with

/\w+@(tickets\.)?company\.com/i

In python, I simply wrote the above regex as '\w+@(tickets\.)?company\.com' expecting the same result. However, support@company.com isn't found at all and a findall on the second returns a list containing only 'tickets.' . So clearly the '(tickets\.)?' is the problem area, but what exactly is the difference in regular expression rules between Perl and Python that I'm missing?

Axeman ,Apr 16, 2009 at 21:10

The documentation for re.findall :
findall(pattern, string, flags=0)
    Return a list of all non-overlapping matches in the string.

    If one or more groups are present in the pattern, return a
    list of groups; this will be a list of tuples if the pattern
    has more than one group.

    Empty matches are included in the result.

Since (tickets\.) is a group, findall returns that instead of the whole match. If you want the whole match, put a group around the whole pattern and/or use non-grouping matches, i.e.

r'(\w+@(tickets\.)?company\.com)'
r'\w+@(?:tickets\.)?company\.com'

Note that you'll have to pick out the first element of each tuple returned by findall in the first case.

chaos ,Apr 16, 2009 at 18:45

I think the problem is in your expectations of extracted values. Try using this in your current Python code:
'(\w+@(?:tickets\.)?company\.com)'

Jason Coon ,Apr 16, 2009 at 18:46

Two problems jump out at me:
  1. You need to use a raw string to avoid having to escape " \ "
  2. You need to escape " . "

So try:

r'\w+@(tickets\.)?company\.com'

EDIT

Sample output:

>>> import re
>>> exp = re.compile(r'\w+@(tickets\.)?company\.com')
>>> bool(exp.match("s@company.com"))
True
>>> bool(exp.match("1234567@tickets.company.com"))
True

,

There isn't a difference in the regexes, but there is a difference in what you are looking for. Your regex is capturing only "tickets." if it exists in both regexes. You probably want something like this
#!/usr/bin/python

import re

regex = re.compile("(\w+@(?:tickets\.)?company\.com)");

a = [
    "foo@company.com", 
    "foo@tickets.company.com", 
    "foo@ticketsacompany.com",
    "foo@compant.org"
];

for string in a:
    print regex.findall(string)

[Oct 22, 2019] Python for a Perl programmer

Oct 22, 2019 | stackoverflow.com

Ask Question Asked 9 years, 8 months ago Active 11 months ago Viewed 22k times 53 47


Hamish Grubijan ,Feb 17, 2010 at 17:56

I am an experienced Perl developer with some degree of experience and/or familiarity with other languages (working experience with C/C++, school experience with Java and Scheme, and passing familiarity with many others).

I might need to get some web work done in Python (most immediately, related to Google App Engine). As such, I'd like to ask SO overmind for good references on how to best learn Python for someone who's coming from Perl background (e.g. the emphasis would be on differences between the two and how to translate perl idiomatics into Python idiomatics, as opposed to generic Python references). Something also centered on Web development is even better. I'll take anything - articles, tutorials, books, sample apps?

Thanks!

FMc ,Dec 19, 2014 at 17:50

I've recently had to make a similar transition for work reasons, and it's been pretty painful. For better or worse, Python has a very different philosophy and way of working than Perl, and getting used to that can be frustrating. The things I've found most useful have been

Personally, I found Dive Into Python annoying and patronising, but it's freely available online, so you can form your own judgment on that.

Philip Durbin ,Feb 18, 2010 at 18:12

If you happen to be a fan of The Perl Cookbook , you might be interested in checking out PLEAC, the Programming Language Examples Alike Cookbook , specifically the section that shows the Perl Cookbook code translated into Python .

larley ,Feb 18, 2010 at 6:16

Being a hardcore Perl programmer, all I can say is DO NOT BUY O'Reilly's "Learning Python". It is nowhere NEAR as good as "Learning Perl", and there's no equivalent I know of to Larry Wall's "Programming Perl", which is simply unbeatable.

I've had the most success taking past Perl programs and translating them into Python, trying to make use of as many new techniques as possible.

Mike Graham ,Feb 17, 2010 at 18:02

Check out the official tutorial , which is actually pretty good. If you are interested in web development you should be ready at that point to jump right in to the documentation of the web framework you will be working with; Python has many to choose from, with zope, cherrypy, pylons, and werkzeug all having good reputations.

I would not try to search for things specifically meant to help you transition from Perl, which are not to be of as high of quality as references that can be useful for more people.

ghostdog74 ,Feb 18, 2010 at 1:17

This is the site you should really go to. There's a section called Getting Started which you should take a look. There are also recommendations on books. On top of that, you might also be interested in this on "idioms"

sateesh ,Feb 17, 2010 at 18:08

If what you are looking at is succinct, concise reference to python then the book Python Essential Reference might be helpful.

Robert P ,May 31, 2013 at 22:39

I wouldn't try to compare Perl and Python too much in order to learn Python, especially since you have working knowledge of other languages. If you are unfamiliar with OOP/Functional programming aspects and just looking to work procedurally like in Perl, start learning the Python language constructs / syntax and then do a couple examples. if you are making a switch to OO or functional style paradigms, I would read up on OO fundamentals first, then start on Python syntax and examples...so you have a sort of mental blueprint of how things can be constructed before you start working with the actual materials. this is just my humble opinion however..

[Oct 21, 2019] Differences between Perl and PHP [closed]

Notable quotes:
"... Perl has native regular expression support, ..."
"... Perl has quite a few more operators , including matching ..."
"... In PHP, new is an operator. In Perl, it's the conventional name of an object creation subroutine defined in packages, nothing special as far as the language is concerned. ..."
"... Perl logical operators return their arguments, while they return booleans in PHP. ..."
"... Perl gives access to the symbol table ..."
"... Note that "references" has a different meaning in PHP and Perl. In PHP, references are symbol table aliases. In Perl, references are smart pointers. ..."
"... Perl has different types for integer-indexed collections (arrays) and string indexed collections (hashes). In PHP, they're the same type: an associative array/ordered map ..."
"... Perl arrays aren't sparse ..."
"... Perl supports hash and array slices natively, ..."
Nov 23, 2013 | stackoverflow.com

jholster ,Nov 23, 2013 at 21:20

I'm planning to learn Perl 5 and as I have only used PHP until now, I wanted to know a bit about how the languages differ from each other.

As PHP started out as a set of "Perl hacks" it has obviously cloned some of Perls features.

hobbs ,Jan 17, 2013 at 8:36

Perl and PHP are more different than alike. Let's consider Perl 5, since Perl 6 is still under development. Some differences, grouped roughly by subject:

PHP was inspired by Perl the same way Phantom of the Paradise was inspired by Phantom of the Opera , or Strange Brew was inspired by Hamlet . It's best to put the behavior specifics of PHP out of your mind when learning Perl, else you'll get tripped up.

My brain hurts now, so I'm going to stop.

Your Common Sense ,Mar 29, 2010 at 2:19

When PHP came to the scene, everyone were impressed with main differences from Perl:
  1. Input variables already in the global scope, no boring parsing.
  2. HTML embedding. Just <?php ... ?> anywhere. No boring templates.
  3. On-screen error messages. No boring error log peeks.
  4. Easy to learn. No boring book reading.

As the time passed, everyone learned that they were not a benefit, hehe...

Quentin ,Jan 15, 2016 at 3:27

I've noticed that most PHP vs. Perl pages seem to be of the

PHP is better than Perl because <insert lame reason here>

ilk, and rarely make reasonable comparisons.

Syntax-wise, you will find PHP is often easier to understand than Perl, particularly when you have little experience. For example, trimming a string of leading and trailing whitespace in PHP is simply

$string = trim($string);

In Perl it is the somewhat more cryptic

$string =~ s/^\s+//;
$string =~ s/\s+$//;

(I believe this is slightly more efficient than a single line capture and replace, and also a little more understandable.) However, even though PHP is often more English-like, it sometimes still shows its roots as a wrapper for low level C, for example, strpbrk and strspn are probably rarely used, because most PHP dabblers write their own equivalent functions for anything too esoteric, rather than spending time exploring the manual. I also wonder about programmers for whom English is a second language, as everybody is on equal footing with things such as Perl, having to learn it from scratch.

I have already mentioned the manual. PHP has a fine online manual, and unfortunately it needs it. I still refer to it from time to time for things that should be simple, such as order of parameters or function naming convention. With Perl, you will probably find you are referring to the manual a lot as you get started and then one day you will have an a-ha moment and never need it again. Well, at least not until you're more advanced and realize that not only is there more than one way, there is probably a better way, somebody else has probably already done it that better way, and perhaps you should just visit CPAN.

Perl does have a lot more options and ways to express things. This is not necessarily a good thing, although it allows code to be more readable if used wisely and at least one of the ways you are likely to be familiar with. There are certain styles and idioms that you will find yourself falling into, and I can heartily recommend reading Perl Best Practices (sooner rather than later), along with Perl Cookbook, Second Edition to get up to speed on solving common problems.

I believe the reason Perl is used less often in shared hosting environments is that historically the perceived slowness of CGI and hosts' unwillingness to install mod_perl due to security and configuration issues has made PHP a more attractive option. The cycle then continued, more people learned to use PHP because more hosts offered it, and more hosts offered it because that's what people wanted to use. The speed differences and security issues are rendered moot by FastCGI these days, and in most cases PHP is run out of FastCGI as well, rather than leaving it in the core of the web server.

Whether or not this is the case or there are other reasons, PHP became popular and a myriad of applications have been written in it. For the majority of people who just want an entry-level website with a simple blog or photo gallery, PHP is all they need so that's what the hosts promote. There should be nothing stopping you from using Perl (or anything else you choose) if you want.

At an enterprise level, I doubt you would find too much PHP in production (and please, no-one point at Facebook as a counter-example, I said enterprise level).

Leon Timmermans ,Mar 28, 2010 at 22:15

Perl is used plenty for websites, no less than Python and Ruby for example. That said, PHP is used way more often than any of those. I think the most important factors in that are PHP's ease of deployment and the ease to start with it.

The differences in syntax are too many to sum up here, but generally it is true that it has more ways to express yourself (this is know as TIMTWOTDI, There Is More Than One Way To Do It).

Brad Gilbert ,Mar 29, 2010 at 4:04

My favorite thing about Perl is the way it handles arrays/lists. Here's an example of how you would make and use a Perl function (or "subroutine"), which makes use of this for arguments:
sub multiply
{
    my ($arg1, $arg2) = @_; # @_ is the array of arguments
    return $arg1 * $arg2;
}

In PHP you could do a similar thing with list() , but it's not quite the same; in Perl lists and arrays are actually treated the same (usually). You can also do things like:

$week_day_name = ("Sunday", "Monday", "Tuesday", "Wednesday", "Thursday", "Friday", "Saturday")[$week_day_index];

And another difference that you MUST know about, is numerical/string comparison operators. In Perl, if you use < , > , == , != , <=> , and so on, Perl converts both operands to numbers. If you want to convert as strings instead, you have to use lt , gt , eq , ne , cmp (the respective equivalents of the operators listed previously). Examples where this will really get you:

if ("a" == "b") { ... } # This is true.
if ("a" == 0) { ... } # This is also true, for the same reason.

Sorin Postelnicu, Aug 5, 2015 at 15:44

I do not need add anything to outis's fantastic answer, i want only show the answer for you question:

Why is Perl not used for dynamic websites very often anymore? What made PHP gain more popularity than it?

Please check first some "Job Trends" sites - and you can make the judgement alone.

as you can see, perl is still a leader - but preferable for real applications not for toys. :)

[Oct 21, 2019] Python variable reference assignment

Oct 21, 2019 | stackoverflow.com

Python variable reference assignment Ask Question Asked 7 years, 3 months ago Active 3 years, 11 months ago Viewed 54k times 42 20


viji ,Jun 27, 2012 at 8:43

In the code
y = 7
x = y    
x = 8

Now, y will be 7 and x will be 8. But actually I wanna change y. Can I assign the reference of y and do that ?

For example, in C++ the same thing can be achieved as,

int y = 8;
int &x = y;
x = 9;

Now both y & x will be 9

Daniel Roseman ,Jun 27, 2012 at 9:20

No, you cannot. As other answer point out, you can (ab?)use aliasing of mutable objects to achieve a similar effect. However, that's not the same thing as C++ references, and I want to explain what actually happens to avoid any misconceptions.

You see, in C++ (and other languages), a variable (and object fields, and entries in collections, etc.) is a storage location and you write a value (for instance, an integer, an object, or a pointer) to that location. In this model, references are an alias for a storage location (of any kind) - when you assign to a non-reference variable, you copy a value (even if it's just a pointer, it's still a value) to the storage location; when you assign to a reference, you copy to a storage location somewhere else. Note that you cannot change a reference itself - once it is bound (and it has to as soon as you create one) all assignments to it alter not the reference but whatever is referred to.

In Python (and other languages), a variable (and object fields, and entries in collections, etc.) is a just a name. Values are somewhere else (e.g. sprinkled all over the heap), and a variable refers (not in the sense of C++ references, more like a pointer minus the pointer arithmetic) to a value. Multiple names can refer to the same value (which is generally a good thing). Python (and other languages) calls whatever is needed to refer to a value a reference, despite being pretty unrelated to things like C++ references and pass-by-reference. Assigning to a variable (or object field, or ...) simply makes it refer to another value. The whole model of storage locations does not apply to Python, the programmer never handles storage locations for values. All he stores and shuffles around are Python references, and those are not values in Python, so they cannot be target of other Python references.

All of this is independent of mutability of the value - it's the same for ints and lists, for instance. You cannot take a variable that refers to either, and overwrite the object it points to. You can only tell the object to modify parts of itself - say, change some reference it contains.

Is this a more restrictive model? Perhaps, but it's powerful enough most of the time. And when it isn't you can work around it, either with a custom class like the one given below, or (equivalent, but less obvious) a single-element collection.

class Reference:
    def __init__(self, val):
        self._value = val # just refers to val, no copy

    def get(self):
        return self._value

    def set(self, val):
        self._value = val

That still won't allow you to alias a "regular" variable or object field, but you can have multiple variables referring to the same Reference object (ditto for the mutable-singleton-collection alternative). You just have to be careful to always use .get() / .set() (or [0] ).

rbaleksandar ,Sep 24, 2013 at 19:34

No, Python doesn't have this feature.

If you had a list (or any other mutable object) you could do what you want by mutating the object that both x and y are bound to:

>>> x = [7]
>>> y = x
>>> y[0] = 8
>>> print x
[8]

See it working online: ideone

tauseef_CuriousGuy ,Sep 7, 2017 at 8:10

You should use a mutable object for this.

In python x & y are just references to objects so y = 7 means y points to the object 7 . x=y means x too points to 7 , but as 7 is immutable so changing the value of x simply changes the object 7 and y still remains pointing to 7 .

>>> y = [7]
>>> x = y
>>> x[0] = 8 # here you're changing [7] not x or y, x & y are just references to [7]
>>> y
[8]

,

Alternatively, you could use a self crafted container.
class Value(object):
    def __init__(self, value): self.value = value

y = Value(7)
x = y
x.value = 8
print y.value

[Oct 20, 2019] Using command line arguments to R CMD BATCH

Oct 20, 2019 | stackoverflow.com

Ask Question Asked 6 years, 9 months ago Active 1 year, 7 months ago Viewed 112k times 93 54


Bryce Thomas ,Jan 5, 2013 at 0:26

I have been using R CMD BATCH my_script.R from a terminal to execute an R script. I am now at the point where I would like to pass an argument to the command, but am having some issues getting it working. If I do R CMD BATCH my_script.R blabla then blabla becomes the output file, rather than being interpreted as an argument available to the R script being executed.

I have tried Rscript my_script.R blabla which seems to pass on blabla correctly as an argument, but then I don't get the my_script.Rout output file that I get with R CMD BATCH (I want the .Rout file). While I could redirect the output of a call to Rscript to a file name of my choosing, I would not be getting the R input commands included in the file in the way R CMD BATCH does in the .Rout file.

So, ideally, I'm after a way to pass arguments to an R script being executed via the R CMD BATCH method, though would be happy with an approach using Rscript if there is a way to make it produce a comparable .Rout file.

Bryce Thomas ,Feb 11, 2015 at 20:59

My impression is that R CMD BATCH is a bit of a relict. In any case, the more recent Rscript executable (available on all platforms), together with commandArgs() makes processing command line arguments pretty easy.

As an example, here is a little script -- call it "myScript.R" :

## myScript.R
args <- commandArgs(trailingOnly = TRUE)
rnorm(n=as.numeric(args[1]), mean=as.numeric(args[2]))

And here is what invoking it from the command line looks like

> Rscript myScript.R 5 100
[1]  98.46435 100.04626  99.44937  98.52910 100.78853

Edit:

Not that I'd recommend it, but ... using a combination of source() and sink() , you could get Rscript to produce an .Rout file like that produced by R CMD BATCH . One way would be to create a little R script -- call it RscriptEcho.R -- which you call directly with Rscript. It might look like this:

## RscriptEcho.R
args <- commandArgs(TRUE)
srcFile <- args[1]
outFile <- paste0(make.names(date()), ".Rout")
args <- args[-1]

sink(outFile, split = TRUE)
source(srcFile, echo = TRUE)

To execute your actual script, you would then do:

Rscript RscriptEcho.R myScript.R 5 100
[1]  98.46435 100.04626  99.44937  98.52910 100.78853

which will execute myScript.R with the supplied arguments and sink interleaved input, output, and messages to a uniquely named .Rout .

Edit2:
You can run Rscript verbosely and place the verbose output in a file.

Rscript --verbose myScript.R 5 100 > myScript.Rout

d2a2d ,Apr 9 at 22:33

After trying the options described here, I found this post from Forester in r-bloggers . I think it is a clean option to consider.

I put his code here:

From command line

$ R CMD BATCH --no-save --no-restore '--args a=1 b=c(2,5,6)' test.R test.out &

Test.R

##First read in the arguments listed at the command line
args=(commandArgs(TRUE))

##args is now a list of character vectors
## First check to see if arguments are passed.
## Then cycle through each element of the list and evaluate the expressions.
if(length(args)==0){
    print("No arguments supplied.")
    ##supply default values
    a = 1
    b = c(1,1,1)
}else{
    for(i in 1:length(args)){
      eval(parse(text=args[[i]]))
    }
}

print(a*2)
print(b*3)

In test.out

> print(a*2)
[1] 2
> print(b*3)
[1]  6 15 18

Thanks to Forester !

user1563570 ,Apr 5, 2013 at 13:15

In your R script, called test.R :
args <- commandArgs(trailingOnly = F)
myargument <- args[length(args)]
myargument <- sub("-","",myargument)
print(myargument)
q(save="no")

From the command line run:

R CMD BATCH -4 test.R

Your output file, test.Rout, will show that the argument 4 has been successfully passed to R:

cat test.Rout

> args <- commandArgs(trailingOnly = F)
> myargument <- args[length(args)]
> myargument <- sub("-","",myargument)
> print(myargument)
[1] "4"
> q(save="no")
> proc.time()
user  system elapsed 
0.222   0.022   0.236

Bryce Thomas ,Oct 23, 2014 at 20:57

You need to put arguments before my_script.R and use - on the arguments, e.g.
R CMD BATCH -blabla my_script.R

commandArgs() will receive -blabla as a character string in this case. See the help for details:

$ R CMD BATCH --help
Usage: R CMD BATCH [options] infile [outfile]

Run R non-interactively with input from infile and place output (stdout
and stderr) to another file.  If not given, the name of the output file
is the one of the input file, with a possible '.R' extension stripped,
and '.Rout' appended.

Options:
  -h, --help        print short help message and exit
  -v, --version     print version info and exit
  --no-timing           do not report the timings
  --            end processing of options

Further arguments starting with a '-' are considered as options as long
as '--' was not encountered, and are passed on to the R process, which
by default is started with '--restore --save --no-readline'.
See also help('BATCH') inside R.

ClementWalter ,Mar 16, 2016 at 9:52

I add an answer because I think a one line solution is always good! Atop of your myRscript.R file, add the following line:
eval(parse(text=paste(commandArgs(trailingOnly = TRUE), collapse=";")))

Then submit your script with something like:

R CMD BATCH [options] '--args arguments you want to supply' myRscript.R &

For example:

R CMD BATCH --vanilla '--args N=1 l=list(a=2, b="test") name="aname"' myscript.R &

Then:

> ls()
[1] "N"    "l"    "name"

Dagremu ,Dec 15, 2016 at 0:50

Here's another way to process command line args, using R CMD BATCH . My approach, which builds on an earlier answer here , lets you specify arguments at the command line and, in your R script, give some or all of them default values.

Here's an R file, which I name test.R :

defaults <- list(a=1, b=c(1,1,1)) ## default values of any arguments we might pass

## parse each command arg, loading it into global environment
for (arg in commandArgs(TRUE))
  eval(parse(text=arg))

## if any variable named in defaults doesn't exist, then create it
## with value from defaults
for (nm in names(defaults))
  assign(nm, mget(nm, ifnotfound=list(defaults[[nm]]))[[1]])

print(a)
print(b)

At the command line, if I type

R CMD BATCH --no-save --no-restore '--args a=2 b=c(2,5,6)' test.R

then within R we'll have a = 2 and b = c(2,5,6) . But I could, say, omit b , and add in another argument c :

R CMD BATCH --no-save --no-restore '--args a=2 c="hello"' test.R

Then in R we'll have a = 2 , b = c(1,1,1) (the default), and c = "hello" .

Finally, for convenience we can wrap the R code in a function, as long as we're careful about the environment:

## defaults should be either NULL or a named list
parseCommandArgs <- function(defaults=NULL, envir=globalenv()) {
  for (arg in commandArgs(TRUE))
    eval(parse(text=arg), envir=envir)

  for (nm in names(defaults))
    assign(nm, mget(nm, ifnotfound=list(defaults[[nm]]), envir=envir)[[1]], pos=envir)
}

## example usage:
parseCommandArgs(list(a=1, b=c(1,1,1)))

[Oct 20, 2019] How can I use multiple library paths?

Oct 20, 2019 | stackoverflow.com

Ask Question Asked 2 years, 10 months ago Active 2 years, 10 months ago Viewed 747 times 2 0

user797963 ,Nov 28, 2016 at 16:11

I'm trying to set up an easy to use R development environment for multiple users. R is installed along with a set of other dev tools on an NFS mount.

I want to create a core set of R packages that also live on NFS so n users don't need to install their own copies of the same packages n times. Then, I was hoping users can install one off packages to a local R library. Has anyone worked with an R setup like this before? From the doc, it looks doable by adding both the core package and personal package file paths to .libPaths() .

,

You want to use the .Renviron file (see ?Startup ).

There are three places to put the file:

In this file you can specify R_LIBS and the R_LIBS_SITE environment variables.

For your particular problem, you probably want to add the NFS drive location to R_LIBS_SITE in the R_HOME/etc/Renviron.site file.


## To get R_HOME
Sys.getenv("R_HOME")

[Oct 20, 2019] How to update a package in R?

Oct 20, 2019 | stackoverflow.com

Ask Question Asked 5 years, 8 months ago Active 10 months ago Viewed 23k times 23 4


Joshua Ulrich ,Jan 31, 2014 at 8:15

I would like to upgrade one R package to the newer version which is already available. I tried
update.packages(c("R2jags"))

but it does nothing! No output on console, no error, nothing. I used the same syntax as for install.packages but perhaps I'm doing something wrong. I have been looking at ?update.packages but I was not been able to figure out how it works, where to specify the package(s) etc. There is no example. I also tried to update the package using install.packages to "install" it again but that says "Warning: package 'R2jags' is in use and will not be installed" .

TMS ,Jan 30, 2014 at 16:47

You can't do this I'm afraid, well, not with update.packages() . You need to call install.packages("R2jags") instead.

You can't install R2jags in the current session because you have already loaded the current version into the session. If you need to, save any objects you can't easily recreate, and quit out of R. Then start a new R session, immediately run install.packages("R2jags") , then once finished, load the package and reload in any previously saved objects. You could try to unload the package with:

detach(package:R2jags, unload = TRUE)

but it is quite complex to do this cleanly unless the package cleans up after itself.

update.packages() exists to update all outdated packages in a stated library location. That library location is given by the first argument (if not supplied it works on all known library locations for the current R session). Hence you were asking it the update the packages in library location R2jags which is most unlikely to exist on your R installation.

amzu ,Jan 30, 2014 at 16:36

Additionally, you can install RStudio and update all packages by going to the Tools menu and selecting Check for Package Updates .

DJ6968 ,Dec 13, 2018 at 11:42

# The following two commands remove any previously installed H2O packages for R.
if ("package:h2o" %in% search()) { detach("package:h2o", unload=TRUE) }
if ("h2o" %in% rownames(installed.packages())) { remove.packages("h2o") }

# Next, we download packages that H2O depends on.
pkgs <- c("RCurl","jsonlite")
for (pkg in pkgs) {
if (! (pkg %in% rownames(installed.packages()))) { install.packages(pkg) }
}

# Now we download, install and initialize the H2O package for R.
install.packages("h2o", type="source", repos="http://h2o-release.s3.amazonaws.com/h2o/rel-xia/2/R")

# Finally, let's load H2O and start up an H2O cluster
library(h2o)`enter code here`
h2o.init()

[Oct 16, 2019] R libraries installation - Stack Overflow

Oct 31, 2016 | stackoverflow.com

R libraries installation Ask Question Asked 2 years, 11 months ago Active 2 years, 10 months ago Viewed 783 times 0

horseshoe ,Oct 31, 2016 at 14:47

I am using a computer where I have read only rights for the R library folder. When I am installing new packages I therefore use
libpath <- "c:/R/mylibraries"
.libPaths( c( .libPaths(), libpath) )
install.packages("htmltools",   lib=libpath)

always when I am installing a new packages with dependecies (like e.g. htmltools depends on lme4), I get erros like:

Error in .requirePackage(package) : 
  unable to find required package 'lme4'

although lme4 is installed and I used it before.... also other errors/warnings like:

Warning in install.packages :
  cannot remove prior installation of package 'Rcpp'

or:

Warning in install.packages :
  unable to move temporary installation 'c:\...\file17b033a54a21\jsonlite' to 'c:\...\jsonlite'

occur. If I install them twice they usually work but sometimes dependencies to packages that worked before are lost and I have to reinstall them again. Is there a way to circumvent this?

> ,

Put this in a file named .REnviron in your Documents folder and restart R:
R_LIBS=c:/R/mylibraries

From then on, you should be able to install packages into that location automatically, without having to fiddle around with .libPaths .

[Oct 16, 2019] R libraries installation - Stack Overflow

Oct 16, 2019 | stackoverflow.com

R libraries installation Ask Question Asked 2 years, 11 months ago Active 2 years, 10 months ago Viewed 783 times 0

horseshoe ,Oct 31, 2016 at 14:47

I am using a computer where I have read only rights for the R library folder. When I am installing new packages I therefore use
libpath <- "c:/R/mylibraries"
.libPaths( c( .libPaths(), libpath) )
install.packages("htmltools",   lib=libpath)

always when I am installing a new packages with dependecies (like e.g. htmltools depends on lme4), I get erros like:

Error in .requirePackage(package) : 
  unable to find required package 'lme4'

although lme4 is installed and I used it before.... also other errors/warnings like:

Warning in install.packages :
  cannot remove prior installation of package 'Rcpp'

or:

Warning in install.packages :
  unable to move temporary installation 'c:\...\file17b033a54a21\jsonlite' to 'c:\...\jsonlite'

occur. If I install them twice they usually work but sometimes dependencies to packages that worked before are lost and I have to reinstall them again. Is there a way to circumvent this?

> ,

Put this in a file named .REnviron in your Documents folder and restart R:
R_LIBS=c:/R/mylibraries

From then on, you should be able to install packages into that location automatically, without having to fiddle around with .libPaths .

[Oct 16, 2019] R setting library path via R_LIBS

Oct 16, 2019 | stackoverflow.com

Ask Question Asked 3 years, 4 months ago Active 3 years, 1 month ago Viewed 5k times 2

Xiongbing Jin ,Jun 12, 2016 at 12:42

I have read the R FAQS and other posts but I am a bit confused and would be grateful to know whether I did everything correctly.

In Windows, in order to modify the default library folder I created a file Renviron.site and put inside E:/Programs/R-3.3.0/etc . The file has only one line saying

R_LIBS=E:/Rlibrary

When I open R and run .libPaths() I see E:/Rlibrary as [1] and the default R library E:/Programs/R-3.3.0/library as [2].

This should mean that from now on all packages I will install will go in E:/Rlibrary but at the same time I will be able to load and use both packages in this folder and those in the default location. Am I correct?

,

When you load a package via library , it will go through each directory in .libPaths() in turn to find the required package. If the package hasn't been found, you will get an error. This means you can have multiple versions of a package (in different directories), but the package that will be used is determined by the order of .libPaths() .

Regarding how .libPaths() is constructed, from ?.R_LIBS

The library search path is initialized at startup from the environment variable 'R_LIBS' (which should be a colon-separated list of directories at which R library trees are rooted) followed by those in environment variable 'R_LIBS_USER'. Only directories which exist at the time will be included.

[Oct 16, 2019] The .Rprofile file

Oct 16, 2019 | csgillespie.github.io

3.3 R startup

Every time R starts, a number of files are read, in a particular order. The contents of these files determine how R performs for the duration of the session. Note that these files should only be changed with caution, as they may make your R version behave differently to other R installations. This could reduce the reproducibility of your code.

Files in three folders are important in this process:

It is important to know the location of the .Rprofile and .Renviron set-up files that are being used out of these three options. R only uses one .Rprofile and one .Renviron in any session: if you have a .Rprofile file in your current project, R will ignore .Rprofile in R_HOME and HOME . Likewise, .Rprofile in HOME overrides .Rprofile in R_HOME . The same applies to .Renviron : you should remember that adding project specific environment variables with .Renviron will de-activate other .Renviron files.

To create a project-specific start-up script, simply create a .Rprofile file in the project's root directory and start adding R code, e.g. via file.edit(".Rprofile") . Remember that this will make .Rprofile in the home directory be ignored. The following commands will open your .Rprofile from within an R editor:

file.edit(file.path("~", ".Rprofile")) # edit .Rprofile in HOME
file.edit(".Rprofile") # edit project specific .Rprofile

Note that editing the .Renviron file in the same locations will have the same effect. The following code will create a user specific .Renviron file (where API keys and other cross-project environment variables can be stored), without overwriting any existing file.

user_renviron = path.expand(file.path("~", ".Renviron"))
if(!file.exists(user_renviron)) # check to see if the file already exists
  file.create(user_renviron)
file.edit(user_renviron) # open with another text editor if this fails

The location, contents and uses of each is outlined in more detail below. 3.3.1 The .Rprofile file

By default R looks for and runs .Rprofile files in the three locations described above, in a specific order. .Rprofile files are simply R scripts that run each time R runs and they can be found within R_HOME , HOME and the project's home directory, found with getwd() . To check if you have a site-wide .Rprofile , which will run for all users on start-up, run:

site_path = R.home(component = "home")
fname = file.path(site_path, "etc", "Rprofile.site")
file.exists(fname)

The above code checks for the presence of Rprofile.site in that directory. As outlined above, the .Rprofile located in your home directory is user-specific. Again, we can test whether this file exists using

file.exists("~/.Rprofile")

We can use R to create and edit .Rprofile (warning: do not overwrite your previous .Rprofile - we suggest you try project-specific .Rprofile first):

if(!file.exists("~/.Rprofile")) # only create if not already there
  file.create("~/.Rprofile")    # (don't overwrite it)
file.edit("~/.Rprofile")
3.3.2 Example .Rprofile settings

An .Rprofile file is just an R script that is run at start-up. The examples at the bottom of the .Rprofile help file

help("Rprofile")

give clues as to the types of things we could place in our profile.

3.3.2.1 Setting options

The function options is a list that contains a number of default options. See help("options") or simply type options() to get an idea of what we can configure. In my .Rprofile file, we have the line

options(prompt="R> ", digits=4, show.signif.stars=FALSE)

This changes three features.

Typically we want to avoid adding options to the start-up file that make our code non-portable. For example, adding

options(stringsAsFactors=FALSE)

to your start-up script has knock-on effects for read.table and related functions including read.csv , making them convert text strings into characters rather than into factors as is default. This may be useful for you, but it is dangerous as it may make your code less portable. 3.3.2.2 Setting the CRAN mirror

To avoid setting the CRAN mirror each time you run install.packages you can permanently set the mirror in your .Rprofile .

## local creates a new, empty environment
## This avoids polluting the global environment with
## the object r
local({
  r = getOption("repos")             
  r["CRAN"] = "https://cran.rstudio.com/"
  options(repos = r)
})

The RStudio mirror is a virtual machine run by Amazon's EC2 service, and it syncs with the main CRAN mirror in Austria once per day. Since RStudio is using Amazon's CloudFront, the repository is automatically distributed around the world, so no matter where you are in the world, the data doesn't need to travel very far, and is therefore fast to download. 3.3.2.3 The fortunes package

This section illustrate what .Rprofile does with reference to a package that was developed for fun. The code below could easily be altered to automatically connect to a database, or ensure that the latest packages have been downloaded.

The fortunes package contains a number of memorable quotes that the community has collected over many years, called R fortunes. Each fortune has a number. To get fortune number , for example, enter

fortunes::fortune(50)

It is easy to make R print out one of these nuggets of truth each time you start a session, by adding the following to ~/.Rprofile :

if(interactive()) 
  try(fortunes::fortune(), silent=TRUE)

The interactive function tests whether R is being used interactively in a terminal. The fortune function is called within try . If the fortunes package is not available, we avoid raising an error and move on. By using :: we avoid adding the fortunes package to our list of attached packages..

The function .Last , if it exists in the .Rprofile , is always run at the end of the session. We can use it to install the fortunes package if needed. To load the package, we use require , since if the package isn't installed, the require function returns FALSE and raises a warning.

.Last = function() {
  cond = suppressWarnings(!require(fortunes, quietly=TRUE))
  if(cond) 
    try(install.packages("fortunes"), silent=TRUE)
  message("Goodbye at ", date(), "\n")
}
3.3.2.4 Useful functions

You can also load useful functions in .Rprofile . For example, we could load the following two functions for examining data frames:

## ht == headtail
ht = function(d, n=6) rbind(head(d, n), tail(d, n))
  
## Show the first 5 rows & first 5 columns of a data frame
hh = function(d) d[1:5, 1:5]

and a function for setting a nice plotting window:

setnicepar = function(mar = c(3, 3, 2, 1), mgp = c(2, 0.4, 0), 
                      tck = -.01, cex.axis = 0.9, 
                      las = 1, mfrow = c(1, 1), ...) {
    par(mar = mar, mgp = mgp, tck = tck,
        cex.axis = cex.axis, las = las, 
        mfrow = mfrow, ...)
}

Note that these functions are for personal use and are unlikely to interfere with code from other people. For this reason even if you use a certain package every day, we don't recommend loading it in your .Rprofile . Also beware the dangers of loading many functions by default: it may make your code less portable. Another downside of putting functions in your .Rprofile is that it can clutter-up your work space: when you run the ls() command, your .Rprofile functions will appear. Also if you run rm(list=ls()) , your functions will be deleted.

One neat trick to overcome this issue is to use hidden objects and environments. When an object name starts with . , by default it doesn't appear in the output of the ls() function

.obj = 1
".obj" %in% ls()
## [1] FALSE

This concept also works with environments. In the .Rprofile file we can create a hidden environment

.env = new.env()

and then add functions to this environment

.env$ht = function(d, n = 6) rbind(head(d, n), tail(d, n))

At the end of the .Rprofile file, we use attach , which makes it possible to refer to objects in the environment by their names alone.

attach(.env)
3.3.3 The .Renviron file

The .Renviron file is used to store system variables. It follows a similar start up routine to the .Rprofile file: R first looks for a global .Renviron file, then for local versions. A typical use of the .Renviron file is to specify the R_LIBS path

## Linux
R_LIBS=~/R/library

## Windows
R_LIBS=C:/R/library

This variable points to a directory where R packages will be installed. When install.packages is called, new packages will be stored in R_LIBS .

Another common use of .Renviron is to store API keys that will be available from one session to another. 4 The following line in .Renviron , for example, sets the ZEIT_KEY environment variable which is used in the package diezeit package:

ZEIT_KEY=PUT_YOUR_KEY_HERE

You will need to sign-in and start a new R session for the environment variable (accessed by Sys.getenv ) to be visible. To test if the example API key has been successfully added as an environment variable, run the following:

Sys.getenv("ZEIT_KEY")

Use of the .Renviron file for storing settings such as library paths and API keys is efficient because it reduces the need to update your settings for every R session. Furthermore, the same .Renviron file will work across different platforms so keep it stored safely. 3.3.4 Exercises

  1. What are the three locations where they are stored? Where are these locations on your computer?
  2. For each location, does a .Rprofile or .Renviron file exist?
  3. Create a .Rprofile file in your current working directory that prints the message Happy efficient R programming each time you start R at this location.

[Oct 16, 2019] Python Variable Declaration - Stack Overflow

Oct 16, 2019 | stackoverflow.com

Python Variable Declaration Ask Question Asked 7 years, 4 months ago Active 20 days ago Viewed 249k times 73 48


jpaugh ,Feb 28, 2017 at 23:52

Learning Python , and has some basic doubts.

1.I have seen variable declaration (path here) as

class writer:
    path = ""

sometimes, no explicit declaration but initialize through __init__ .

def __init__(self, name):
    self.name = name

I understand the purpose of __init__ , but is it advisable to declare variable in any other functions.

2.How can I create variable to hold a custom type?

class writer:
    path = "" # string value
    customObj = ??

Martijn Pieters ♦ ,Mar 9 at 3:52

Okay, first things first.

There is no such thing as "variable declaration" or "variable initialization" in Python.

There is simply what we call "assignment", but should probably just call "naming".

Assignment means "this name on the left-hand side now refers to the result of evaluating the right-hand side, regardless of what it referred to before (if anything)".

foo = 'bar' # the name 'foo' is now a name for the string 'bar'
foo = 2 * 3 # the name 'foo' stops being a name for the string 'bar',
# and starts being a name for the integer 6, resulting from the multiplication

As such, Python's names (a better term than "variables", arguably) don't have associated types; the values do. You can re-apply the same name to anything regardless of its type, but the thing still has behaviour that's dependent upon its type. The name is simply a way to refer to the value (object). This answers your second question: You don't create variables to hold a custom type. You don't create variables to hold any particular type. You don't "create" variables at all. You give names to objects.

Second point: Python follows a very simple rule when it comes to classes, that is actually much more consistent than what languages like Java, C++ and C# do: everything declared inside the class block is part of the class . So, functions ( def ) written here are methods, i.e. part of the class object (not stored on a per-instance basis), just like in Java, C++ and C#; but other names here are also part of the class. Again, the names are just names, and they don't have associated types, and functions are objects too in Python. Thus:

class Example:
    data = 42
    def method(self): pass

Classes are objects too , in Python.

So now we have created an object named Example , which represents the class of all things that are Example s. This object has two user-supplied attributes (In C++, "members"; in C#, "fields or properties or methods"; in Java, "fields or methods"). One of them is named data , and it stores the integer value 42 . The other is named method , and it stores a function object. (There are several more attributes that Python adds automatically.)

These attributes still aren't really part of the object, though. Fundamentally, an object is just a bundle of more names (the attribute names), until you get down to things that can't be divided up any more. Thus, values can be shared between different instances of a class, or even between objects of different classes, if you deliberately set that up.

Let's create an instance:

x = Example()

Now we have a separate object named x , which is an instance of Example . The data and method are not actually part of the object, but we can still look them up via x because of some magic that Python does behind the scenes. When we look up method , in particular, we will instead get a "bound method" (when we call it, x gets passed automatically as the self parameter, which cannot happen if we look up Example.method directly).

What happens when we try to use x.data ?

When we examine it, it's looked up in the object first. If it's not found in the object, Python looks in the class.

However, when we assign to x.data , Python will create an attribute on the object. It will not replace the class' attribute.

This allows us to do object initialization. Python will automatically call the class' __init__ method on new instances when they are created, if present. In this method, we can simply assign to attributes to set initial values for that attribute on each object:

class Example:
    name = "Ignored"
    def __init__(self, name):
        self.name = name
    # rest as before

Now we must specify a name when we create an Example , and each instance has its own name . Python will ignore the class attribute Example.name whenever we look up the .name of an instance, because the instance's attribute will be found first.

One last caveat: modification (mutation) and assignment are different things!

In Python, strings are immutable. They cannot be modified. When you do:

a = 'hi '
b = a
a += 'mom'

You do not change the original 'hi ' string. That is impossible in Python. Instead, you create a new string 'hi mom' , and cause a to stop being a name for 'hi ' , and start being a name for 'hi mom' instead. We made b a name for 'hi ' as well, and after re-applying the a name, b is still a name for 'hi ' , because 'hi ' still exists and has not been changed.

But lists can be changed:

a = [1, 2, 3]
b = a
a += [4]

Now b is [1, 2, 3, 4] as well, because we made b a name for the same thing that a named, and then we changed that thing. We did not create a new list for a to name, because Python simply treats += differently for lists.

This matters for objects because if you had a list as a class attribute, and used an instance to modify the list, then the change would be "seen" in all other instances. This is because (a) the data is actually part of the class object, and not any instance object; (b) because you were modifying the list and not doing a simple assignment, you did not create a new instance attribute hiding the class attribute.

detly ,Jun 13, 2012 at 3:38

There's no need to declare new variables in Python. If we're talking about variables in functions or modules, no declaration is needed. Just assign a value to a name where you need it: mymagic = "Magic" . Variables in Python can hold values of any type, and you can't restrict that.

Your question specifically asks about classes, objects and instance variables though. The idiomatic way to create instance variables is in the __init__ method and nowhere else -- while you could create new instance variables in other methods, or even in unrelated code, it's just a bad idea. It'll make your code hard to reason about or to maintain.

So for example:

class Thing(object):

    def __init__(self, magic):
        self.magic = magic

Easy. Now instances of this class have a magic attribute:

thingo = Thing("More magic")
# thingo.magic is now "More magic"

Creating variables in the namespace of the class itself leads to different behaviour altogether. It is functionally different, and you should only do it if you have a specific reason to. For example:

class Thing(object):

    magic = "Magic"

    def __init__(self):
        pass

Now try:

thingo = Thing()
Thing.magic = 1
# thingo.magic is now 1

Or:

class Thing(object):

    magic = ["More", "magic"]

    def __init__(self):
        pass

thing1 = Thing()
thing2 = Thing()
thing1.magic.append("here")
# thing1.magic AND thing2.magic is now ["More", "magic", "here"]

This is because the namespace of the class itself is different to the namespace of the objects created from it. I'll leave it to you to research that a bit more.

The take-home message is that idiomatic Python is to (a) initialise object attributes in your __init__ method, and (b) document the behaviour of your class as needed. You don't need to go to the trouble of full-blown Sphinx-level documentation for everything you ever write, but at least some comments about whatever details you or someone else might need to pick it up.

O. Aroesti ,Jul 4, 2018 at 14:35

This might be 6 years late, but in Python 3.5 and above, you declare a variable type like this:
variable_name: type_name

or this:

variable_name # type: shinyType

So in your case(if you have a CustomObject class defined), you can do:

customObj: CustomObject

See this or that for more info.

redhot ,Jun 22, 2017 at 15:29

For scoping purpose, I use:
custom_object = None

,

Variables have scope, so yes it is appropriate to have variables that are specific to your function. You don't always have to be explicit about their definition; usually you can just use them. Only if you want to do something specific to the type of the variable, like append for a list, do you need to define them before you start using them. Typical example of this.
list = []
for i in stuff:
  list.append(i)

By the way, this is not really a good way to setup the list. It would be better to say:

list = [i for i in stuff] # list comprehension

...but I digress.

Your other question. The custom object should be a class itself.

class CustomObject(): # always capitalize the class name...this is not syntax, just style.
  pass
customObj = CustomObject()

[Oct 15, 2019] Perl to Python Function translation [closed]

Feb 01, 2014 | stackoverflow.com

Ask Question Asked 5 years, 8 months ago Active 5 years, 8 months ago Viewed 303 times -3


Jim Garrison ,Feb 1, 2014 at 22:24

I am trying to translate a Perl function into a Python function, but I am having trouble figuring out what some of the Perl to Python function equivalents.

Perl function:

sub reverse_hex {

 my $HEXDATE = shift;
 my @bytearry=();
 my $byte_cnt = 0;
 my $max_byte_cnt = 8;
 my $byte_offset = 0;
 while($byte_cnt < $max_byte_cnt) {
   my $tmp_str = substr($HEXDATE,$byte_offset,2);
    push(@bytearry,$tmp_str);
   $byte_cnt++;
   $byte_offset+=2;
 }
   return join('',reverse(@bytearry));
}

I am not sure what "push", "shift", and "substr" are doing here that would be the same in Python.

Any help will be much appreciated.

Kenosis ,Feb 1, 2014 at 22:17

The Perl subroutine seems rather complicated for what it does, viz., taking chunks of two chars at a time (the first 16 chars) from the sent string and then reverses it. Another Perl option is:
sub reverse_hex {
    return join '', reverse unpack 'A2' x 8, $_[0];
}

First, unpack here takes two characters at a time (eight times) and produces a list. That list is reverse d and join ed to produce the final string.

Here's a Python subroutine to accomplish this:

def reverse_hex(HEXDATE):
    hexVals = [HEXDATE[i:i + 2] for i in xrange(0, 16, 2)]
    reversedHexVals = hexVals[::-1]
    return ''.join(reversedHexVals)

The list comprehension produces eight elements of two characters each. [::-1] reverses the list's elements and the result is join ed and returned.

Hope this helps!

MikeMayer67 ,Feb 2, 2014 at 2:10

I realize that you are asking about the perl to python translation, but if you have any control over the perl, I would like to point out that this function is a lot more complicated than it needs to be.

The entire thing could be replaced with:

sub reverse_hex
{
  my $hexdate = shift;
  my @bytes = $hexdate =~ /../g;  # break $hexdate into array of character pairs
  return join '', reverse(@bytes);
}

Not only is this shorter, it is much easier to get your head around. Of course, if you have no control over the perl, you are stuck with what you were dealt.

[Oct 15, 2019] What's the easiest way to install a missing Perl module

Oct 15, 2019 | stackoverflow.com

ikegami ,Dec 13, 2017 at 21:21

I get this error:

Can't locate Foo.pm in @INC

Is there an easier way to install it than downloading, untarring, making, etc?

brian d foy ,Jun 10, 2014 at 21:52

On Unix :

usually you start cpan in your shell:

# cpan

and type

install Chocolate::Belgian

or in short form:

cpan Chocolate::Belgian

On Windows :

If you're using ActivePerl on Windows, the PPM (Perl Package Manager) has much of the same functionality as CPAN.pm.

Example:

# ppm
ppm> search net-smtp
ppm> install Net-SMTP-Multipart

see How do I install Perl modules? in the CPAN FAQ

Many distributions ship a lot of perl modules as packages.

You should always prefer them as you benefit from automatic (security) updates and the ease of removal . This can be pretty tricky with the cpan tool itself.

For Gentoo there's a nice tool called g-cpan which builds/installs the module from CPAN and creates a Gentoo package ( ebuild ) for you.

Chas. Owens ,Jan 30, 2017 at 21:08

Try App::cpanminus :
# cpanm Chocolate::Belgian

It's great for just getting stuff installed. It provides none of the more complex functionality of CPAN or CPANPLUS, so it's easy to use, provided you know which module you want to install. If you haven't already got cpanminus, just type:

# cpan App::cpanminus

to install it.

It is also possible to install it without using cpan at all. The basic bootstrap procedure is,

curl -L http://cpanmin.us | perl - --sudo App::cpanminus

For more information go to the App::cpanminus page and look at the section on installation.

isomorphismes ,Mar 22, 2011 at 16:03

I note some folks suggesting one run cpan under sudo. That used to be necessary to install into the system directory, but modern versions of the CPAN shell allow you to configure it to use sudo just for installing. This is much safer, since it means that tests don't run as root.

If you have an old CPAN shell, simply install the new cpan ("install CPAN") and when you reload the shell, it should prompt you to configure these new directives.

Nowadays, when I'm on a system with an old CPAN, the first thing I do is update the shell and set it up to do this so I can do most of my cpan work as a normal user.

Also, I'd strongly suggest that Windows users investigate strawberry Perl . This is a version of Perl that comes packaged with a pre-configured CPAN shell as well as a compiler. It also includes some hard-to-compile Perl modules with their external C library dependencies, notably XML::Parser. This means that you can do the same thing as every other Perl user when it comes to installing modules, and things tend to "just work" a lot more often.

Ivan X ,Sep 19 at 19:32

If you're on Ubuntu and you want to install the pre-packaged perl module (for example, geo::ipfree) try this:
    $ apt-cache search perl geo::ipfree
    libgeo-ipfree-perl - A look up country of ip address Perl module

    $ sudo apt-get install libgeo-ipfree-perl

brian d foy ,Sep 15, 2008 at 22:47

A couple of people mentioned the cpan utility, but it's more than just starting a shell. Just give it the modules that you want to install and let it do it's work.
$prompt> cpan Foo::Bar

If you don't give it any arguments it starts the CPAN.pm shell. This works on Unix, Mac, and should be just fine on Windows (especially Strawberry Perl).

There are several other things that you can do with the cpan tool as well. Here's a summary of the current features (which might be newer than the one that comes with CPAN.pm and perl):

-a
Creates the CPAN.pm autobundle with CPAN::Shell->autobundle.

-A module [ module ... ]
Shows the primary maintainers for the specified modules

-C module [ module ... ]
Show the Changes files for the specified modules

-D module [ module ... ]
Show the module details. This prints one line for each out-of-date module (meaning,
modules locally installed but have newer versions on CPAN). Each line has three columns:
module name, local version, and CPAN version.

-L author [ author ... ]
List the modules by the specified authors.

-h
Prints a help message.

-O
Show the out-of-date modules.

-r
Recompiles dynamically loaded modules with CPAN::Shell->recompile.

-v
Print the script version and CPAN.pm version.

wytten ,Apr 25 at 19:26

sudo perl -MCPAN -e 'install Foo'

Corion ,Sep 16, 2008 at 6:36

Also see Yes, even you can use CPAN . It shows how you can use CPAN without having root or sudo access.

mikegrb ,Sep 16, 2008 at 19:25

Otto made a good suggestion . This works for Debian too, as well as any other Debian derivative. The missing piece is what to do when apt-cache search doesn't find something.
$ sudo apt-get install dh-make-perl build-essential apt-file
$ sudo apt-file update

Then whenever you have a random module you wish to install:

$ cd ~/some/path
$ dh-make-perl --build --cpan Some::Random::Module
$ sudo dpkg -i libsome-random-module-perl-0.01-1_i386.deb

This will give you a deb package that you can install to get Some::Random::Module. One of the big benefits here is man pages and sample scripts in addition to the module itself will be placed in your distro's location of choice. If the distro ever comes out with an official package for a newer version of Some::Random::Module, it will automatically be installed when you apt-get upgrade.

community wiki
jm666
,May 22, 2011 at 18:19

Already answered and accepted answer - but anyway:

IMHO the easiest way installing CPAN modules (on unix like systems, and have no idea about the wondows) is:

curl -L http://cpanmin.us | perl - --sudo App::cpanminus

The above is installing the "zero configuration CPAN modules installer" called cpanm . (Can take several minutes to install - don't break the process)

and after - simply:

cpanm Foo
cpanm Module::One
cpanm Another::Module

brian d foy ,Oct 8, 2008 at 7:26

Lots of recommendation for CPAN.pm , which is great, but if you're using Perl 5.10 then you've also got access to CPANPLUS.pm which is like CPAN.pm but better.

And, of course, it's available on CPAN for people still using older versions of Perl. Why not try:

$ cpan CPANPLUS

IgorGanapolsky ,Jan 30, 2017 at 21:09

Many times it does happen that cpan install command fails with the message like "make test had returned bad status, won't install without force"

In that case following is the way to install the module:

perl -MCPAN -e "CPAN::Shell->force(qw(install Foo::Bar));"

community wiki
2 revs, 2 users 80%
,Apr 13, 2015 at 14:50

On ubuntu most perl modules are already packaged, so installing is much faster than most other systems which have to compile.

To install Foo::Bar at a commmand prompt for example usually you just do:

sudo apt-get install libfoo-bar-perl

Sadly not all modules follow that naming convention.

community wiki
3 revs, 3 users 50%
,Apr 13, 2015 at 14:52

Even it should work:
cpan -i module_name

community wiki
3 revs, 2 users 97%
,Apr 13, 2015 at 16:43

Use cpan command as cpan Modulename
$ cpan HTML::Parser

To install dependencies automatically follow the below

$ perl -MCPAN -e shell
cpan[1]>  o conf prerequisites_policy follow
cpan[2]>  o conf commit
exit

I prefer App::cpanminus , it installs dependencies automatically. Just do

$ cpanm HTML::Parser

brian d foy ,Sep 27, 2008 at 18:58

2 ways that I know of :

USING PPM :

With Windows (ActivePerl) I've used ppm

from the command line type ppm. At the ppm prompt ...

ppm> install foo

or

ppm> search foo

to get a list of foo modules available. Type help for all the commands

USING CPAN :

you can also use CPAN like this ( *nix systems ) :

perl -MCPAN -e 'shell'

gets you a prompt

cpan>

at the prompt ...

cpan> install foo  (again to install the foo module)

type h to get a list of commands for cpan

community wiki
Bruce Alderman
,Nov 21, 2008 at 19:59

On Fedora you can use
# yum install foo

as long as Fedora has an existing package for the module.

community wiki
2 revs, 2 users 89%
,Apr 13, 2015 at 14:51

On Fedora Linux or Enterprise Linux , yum also tracks perl library dependencies. So, if the perl module is available, and some rpm package exports that dependency, it will install the right package for you.
yum install 'perl(Chocolate::Belgian)'

(most likely perl-Chocolate-Belgian package, or even ChocolateFactory package)

community wiki
Mister X
,Dec 28, 2016 at 11:16

Easiest way for me is this:
PERL_MM_USE_DEFAULT=1 perl -MCPAN -e 'install DateTime::TimeZone'

a) automatic recursive dependency detection/resolving/installing

b) it's a shell onliner, good for setup-scripts

community wiki
venkrao
,Sep 11, 2013 at 18:06

If you want to put the new module into a custom location that your cpan shell isn't configured to use, then perhaps, the following will be handy.
 #wget <URL to the module.tgz>
 ##unpack
 perl Build.PL
./Build destdir=$HOME install_base=$HOME
./Build destdir=$HOME install_base=$HOME install

community wiki
2 revs, 2 users 67%
,Apr 13, 2015 at 14:50

Sometimes you can use the yum search foo to search the relative perl module, then use yum install xxx to install.

PW. ,Sep 15, 2008 at 19:26

On Windows with the ActiveState distribution of Perl, use the ppm command.

community wiki
Kamal Nayan
,Oct 1, 2015 at 9:56

Simply executing cpan Foo::Bar on shell would serve the purpose.

community wiki
Ed Dunn
,Nov 4, 2016 at 15:26

Seems like you've already got your answer but I figured I'd chime in. This is what I do in some scripts on an Ubuntu (or debian server)
#!/usr/bin/perl

use warnings;
use strict;

#I've gotten into the habit of setting this on all my scripts, prevents weird path issues if the script is not being run by root
$ENV{'PATH'} = '/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin';

#Fill this with the perl modules required for your project
my @perl = qw(LWP::Simple XML::LibXML MIME::Lite DBI DateTime Config::Tiny Proc::ProcessTable);

chomp(my $curl = `which curl`);

if(!$curl){ system('apt-get install curl -y > /dev/null'); }

chomp(my $cpanm = system('/bin/bash', '-c', 'which cpanm &>/dev/null'));

#installs cpanm if missing
if($cpanm){ system('curl -s -L http://cpanmin.us | perl - --sudo App::cpanminus'); }

#loops through required modules and installs them if missing
foreach my $x (@perl){
    eval "use $x";
    if($@){
        system("cpanm $x");
        eval "use $x";
    }
}

This works well for me, maybe there is something here you can use.

[Oct 13, 2019] What is the system function in python

Oct 13, 2019 | stackoverflow.com

what is the system function in python Ask Question Asked 9 years, 3 months ago Active 9 years, 3 months ago Viewed 6k times 1


Eva Feldman ,Jul 6, 2010 at 15:55

I want to play with system command in python . for example we have this function in perl : system("ls -la"); and its run ls -la what is the system function in python ? Thanks in Advance .

Felix Kling ,Jul 6, 2010 at 15:58

It is os.system :
import os
os.system('ls -la')

But this won't give you any output. So subprocess.check_output is probably more what you want:

>>> import subprocess
>>> subprocess.check_output(["ls", "-l", "/dev/null"])
'crw-rw-rw- 1 root root 1, 3 Oct 18  2007 /dev/null\n'

KLee1 ,Jul 6, 2010 at 16:00

import os
os.system("")

From here

> ,

In the os module there is os.system() .

But if you want to do more advanced things with subprocesses the subprocess module provides a higher level interface with more possibilities that is usually preferable.

[Oct 13, 2019] python - Find size and free space of the filesystem containing a given file - Stack Overflow

Oct 13, 2019 | stackoverflow.com

Find size and free space of the filesystem containing a given file Ask Question Asked 8 years, 10 months ago Active 6 months ago Viewed 110k times 67 21


Piskvor ,Aug 21, 2013 at 7:19

I'm using Python 2.6 on Linux. What is the fastest way:

Sven Marnach ,May 5, 2016 at 11:11

If you just need the free space on a device, see the answer using os.statvfs() below.

If you also need the device name and mount point associated with the file, you should call an external program to get this information. df will provide all the information you need -- when called as df filename it prints a line about the partition that contains the file.

To give an example:

import subprocess
df = subprocess.Popen(["df", "filename"], stdout=subprocess.PIPE)
output = df.communicate()[0]
device, size, used, available, percent, mountpoint = \
    output.split("\n")[1].split()

Note that this is rather brittle, since it depends on the exact format of the df output, but I'm not aware of a more robust solution. (There are a few solutions relying on the /proc filesystem below that are even less portable than this one.)

Halfgaar ,Feb 9, 2017 at 10:41

This doesn't give the name of the partition, but you can get the filesystem statistics directly using the statvfs Unix system call. To call it from Python, use os.statvfs('/home/foo/bar/baz') .

The relevant fields in the result, according to POSIX :

unsigned long f_frsize   Fundamental file system block size. 
fsblkcnt_t    f_blocks   Total number of blocks on file system in units of f_frsize. 
fsblkcnt_t    f_bfree    Total number of free blocks. 
fsblkcnt_t    f_bavail   Number of free blocks available to 
                         non-privileged process.

So to make sense of the values, multiply by f_frsize :

import os
statvfs = os.statvfs('/home/foo/bar/baz')

statvfs.f_frsize * statvfs.f_blocks     # Size of filesystem in bytes
statvfs.f_frsize * statvfs.f_bfree      # Actual number of free bytes
statvfs.f_frsize * statvfs.f_bavail     # Number of free bytes that ordinary users
                                        # are allowed to use (excl. reserved space)

Halfgaar ,Feb 9, 2017 at 10:44

import os

def get_mount_point(pathname):
    "Get the mount point of the filesystem containing pathname"
    pathname= os.path.normcase(os.path.realpath(pathname))
    parent_device= path_device= os.stat(pathname).st_dev
    while parent_device == path_device:
        mount_point= pathname
        pathname= os.path.dirname(pathname)
        if pathname == mount_point: break
        parent_device= os.stat(pathname).st_dev
    return mount_point

def get_mounted_device(pathname):
    "Get the device mounted at pathname"
    # uses "/proc/mounts"
    pathname= os.path.normcase(pathname) # might be unnecessary here
    try:
        with open("/proc/mounts", "r") as ifp:
            for line in ifp:
                fields= line.rstrip('\n').split()
                # note that line above assumes that
                # no mount points contain whitespace
                if fields[1] == pathname:
                    return fields[0]
    except EnvironmentError:
        pass
    return None # explicit

def get_fs_freespace(pathname):
    "Get the free space of the filesystem containing pathname"
    stat= os.statvfs(pathname)
    # use f_bfree for superuser, or f_bavail if filesystem
    # has reserved space for superuser
    return stat.f_bfree*stat.f_bsize

Some sample pathnames on my computer:

path 'trash':
  mp /home /dev/sda4
  free 6413754368
path 'smov':
  mp /mnt/S /dev/sde
  free 86761562112
path '/usr/local/lib':
  mp / rootfs
  free 2184364032
path '/proc/self/cmdline':
  mp /proc proc
  free 0
PS

if on Python ≥3.3, there's shutil.disk_usage(path) which returns a named tuple of (total, used, free) expressed in bytes.

Xiong Chiamiov ,Sep 30, 2016 at 20:39

As of Python 3.3, there an easy and direct way to do this with the standard library:
$ cat free_space.py 
#!/usr/bin/env python3

import shutil

total, used, free = shutil.disk_usage(__file__)
print(total, used, free)

$ ./free_space.py 
1007870246912 460794834944 495854989312

These numbers are in bytes. See the documentation for more info.

Giampaolo Rodolà ,Aug 16, 2017 at 9:08

This should make everything you asked:
import os
from collections import namedtuple

disk_ntuple = namedtuple('partition',  'device mountpoint fstype')
usage_ntuple = namedtuple('usage',  'total used free percent')

def disk_partitions(all=False):
    """Return all mountd partitions as a nameduple.
    If all == False return phyisical partitions only.
    """
    phydevs = []
    f = open("/proc/filesystems", "r")
    for line in f:
        if not line.startswith("nodev"):
            phydevs.append(line.strip())

    retlist = []
    f = open('/etc/mtab', "r")
    for line in f:
        if not all and line.startswith('none'):
            continue
        fields = line.split()
        device = fields[0]
        mountpoint = fields[1]
        fstype = fields[2]
        if not all and fstype not in phydevs:
            continue
        if device == 'none':
            device = ''
        ntuple = disk_ntuple(device, mountpoint, fstype)
        retlist.append(ntuple)
    return retlist

def disk_usage(path):
    """Return disk usage associated with path."""
    st = os.statvfs(path)
    free = (st.f_bavail * st.f_frsize)
    total = (st.f_blocks * st.f_frsize)
    used = (st.f_blocks - st.f_bfree) * st.f_frsize
    try:
        percent = ret = (float(used) / total) * 100
    except ZeroDivisionError:
        percent = 0
    # NB: the percentage is -5% than what shown by df due to
    # reserved blocks that we are currently not considering:
    # http://goo.gl/sWGbH
    return usage_ntuple(total, used, free, round(percent, 1))


if __name__ == '__main__':
    for part in disk_partitions():
        print part
        print "    %s\n" % str(disk_usage(part.mountpoint))

On my box the code above prints:

giampaolo@ubuntu:~/dev$ python foo.py 
partition(device='/dev/sda3', mountpoint='/', fstype='ext4')
    usage(total=21378641920, used=4886749184, free=15405903872, percent=22.9)

partition(device='/dev/sda7', mountpoint='/home', fstype='ext4')
    usage(total=30227386368, used=12137168896, free=16554737664, percent=40.2)

partition(device='/dev/sdb1', mountpoint='/media/1CA0-065B', fstype='vfat')
    usage(total=7952400384, used=32768, free=7952367616, percent=0.0)

partition(device='/dev/sr0', mountpoint='/media/WB2PFRE_IT', fstype='iso9660')
    usage(total=695730176, used=695730176, free=0, percent=100.0)

partition(device='/dev/sda6', mountpoint='/media/Dati', fstype='fuseblk')
    usage(total=914217758720, used=614345637888, free=299872120832, percent=67.2)

AK47 ,Jul 7, 2016 at 10:37

The simplest way to find out it.
import os
from collections import namedtuple

DiskUsage = namedtuple('DiskUsage', 'total used free')

def disk_usage(path):
    """Return disk usage statistics about the given path.

    Will return the namedtuple with attributes: 'total', 'used' and 'free',
    which are the amount of total, used and free space, in bytes.
    """
    st = os.statvfs(path)
    free = st.f_bavail * st.f_frsize
    total = st.f_blocks * st.f_frsize
    used = (st.f_blocks - st.f_bfree) * st.f_frsize
    return DiskUsage(total, used, free)

tzot ,Aug 8, 2011 at 10:11

For the first point, you can try using os.path.realpath to get a canonical path, check it against /etc/mtab (I'd actually suggest calling getmntent , but I can't find a normal way to access it) to find the longest match. (to be sure, you should probably stat both the file and the presumed mountpoint to verify that they are in fact on the same device)

For the second point, use os.statvfs to get block size and usage information.

(Disclaimer: I have tested none of this, most of what I know came from the coreutils sources)

andrew ,Dec 15, 2017 at 0:55

For the second part of your question, "get usage statistics of the given partition", psutil makes this easy with the disk_usage(path) function. Given a path, disk_usage() returns a named tuple including total, used, and free space expressed in bytes, plus the percentage usage.

Simple example from documentation:

>>> import psutil
>>> psutil.disk_usage('/')
sdiskusage(total=21378641920, used=4809781248, free=15482871808, percent=22.5)

Psutil works with Python versions from 2.6 to 3.6 and on Linux, Windows, and OSX among other platforms.

Donald Duck ,Jan 12, 2018 at 18:28

import os

def disk_stat(path):
    disk = os.statvfs(path)
    percent = (disk.f_blocks - disk.f_bfree) * 100 / (disk.f_blocks -disk.f_bfree + disk.f_bavail) + 1
    return percent


print disk_stat('/')
print disk_stat('/data')

> ,

Usually the /proc directory contains such information in Linux, it is a virtual filesystem. For example, /proc/mounts gives information about current mounted disks; and you can parse it directly. Utilities like top , df all make use of /proc .

I haven't used it, but this might help too, if you want a wrapper: http://bitbucket.org/chrismiles/psi/wiki/Home

[Oct 09, 2019] scope - What is the difference between my and our in Perl - Stack Overflow

Oct 09, 2019 | stackoverflow.com

Asked 10 years, 5 months ago Active 3 years, 1 month ago Viewed 107k times 180 56


Nathan Fellman ,May 10, 2009 at 10:24

I know what my is in Perl. It defines a variable that exists only in the scope of the block in which it is defined. What does our do? How does our differ from my ?

Nathan Fellman ,Nov 20, 2016 at 1:15

Great question: How does our differ from my and what does our do?

In Summary:

Available since Perl 5, my is a way to declare:


On the other hand, our variables are:


Declaring a variable with our allows you to predeclare variables in order to use them under use strict without getting typo warnings or compile-time errors. Since Perl 5.6, it has replaced the obsolete use vars , which was only file-scoped, and not lexically scoped as is our .

For example, the formal, qualified name for variable $x inside package main is $main::x . Declaring our $x allows you to use the bare $x variable without penalty (i.e., without a resulting error), in the scope of the declaration, when the script uses use strict or use strict "vars" . The scope might be one, or two, or more packages, or one small block.

Georg ,Oct 1, 2016 at 6:41

The PerlMonks and PerlDoc links from cartman and Olafur are a great reference - below is my crack at a summary:

my variables are lexically scoped within a single block defined by {} or within the same file if not in {} s. They are not accessible from packages/subroutines defined outside of the same lexical scope / block.

our variables are scoped within a package/file and accessible from any code that use or require that package/file - name conflicts are resolved between packages by prepending the appropriate namespace.

Just to round it out, local variables are "dynamically" scoped, differing from my variables in that they are also accessible from subroutines called within the same block.

Nathan Fellman ,Nov 20, 2015 at 18:46

An example:
use strict;

for (1 .. 2){
    # Both variables are lexically scoped to the block.
    our ($o);  # Belongs to 'main' package.
    my  ($m);  # Does not belong to a package.

    # The variables differ with respect to newness.
    $o ++;
    $m ++;
    print __PACKAGE__, " >> o=$o m=$m\n";  # $m is always 1.

    # The package has changed, but we still have direct,
    # unqualified access to both variables, because the
    # lexical scope has not changed.
    package Fubb;
    print __PACKAGE__, " >> o=$o m=$m\n";
}

# The our() and my() variables differ with respect to privacy.
# We can still access the variable declared with our(), provided
# that we fully qualify its name, but the variable declared
# with my() is unavailable.
print __PACKAGE__, " >> main::o=$main::o\n";  # 2
print __PACKAGE__, " >> main::m=$main::m\n";  # Undefined.

# Attempts to access the variables directly won't compile.
# print __PACKAGE__, " >> o=$o\n";
# print __PACKAGE__, " >> m=$m\n";

# Variables declared with use vars() are like those declared
# with our(): belong to a package; not private; and not new.
# However, their scoping is package-based rather than lexical.
for (1 .. 9){
    use vars qw($uv);
    $uv ++;
}

# Even though we are outside the lexical scope where the
# use vars() variable was declared, we have direct access
# because the package has not changed.
print __PACKAGE__, " >> uv=$uv\n";

# And we can access it from another package.
package Bubb;
print __PACKAGE__, " >> main::uv=$main::uv\n";

daotoad ,May 10, 2009 at 16:37

Coping with Scoping is a good overview of Perl scoping rules. It's old enough that our is not discussed in the body of the text. It is addressed in the Notes section at the end.

The article talks about package variables and dynamic scope and how that differs from lexical variables and lexical scope.

Chas. Owens ,Oct 7, 2013 at 14:02

my is used for local variables, where as our is used for global variables. More reading over Variable Scoping in Perl: the basics .

ruffin ,Feb 10, 2015 at 19:47

It's an old question, but I ever met some pitfalls about lexical declarations in Perl that messed me up, which are also related to this question, so I just add my summary here:

1. definition or declaration?

local $var = 42; 
print "var: $var\n";

The output is var: 42 . However we couldn't tell if local $var = 42; is a definition or declaration. But how about this:

use strict;
use warnings;

local $var = 42;
print "var: $var\n";

The second program will throw an error:

Global symbol "$var" requires explicit package name.

$var is not defined, which means local $var; is just a declaration! Before using local to declare a variable, make sure that it is defined as a global variable previously.

But why this won't fail?

use strict;
use warnings;

local $a = 42;
print "var: $a\n";

The output is: var: 42 .

That's because $a , as well as $b , is a global variable pre-defined in Perl. Remember the sort function?

2. lexical or global?

I was a C programmer before starting using Perl, so the concept of lexical and global variables seems straightforward to me: just corresponds to auto and external variables in C. But there're small differences:

In C, an external variable is a variable defined outside any function block. On the other hand, an automatic variable is a variable defined inside a function block. Like this:

int global;

int main(void) {
    int local;
}

While in Perl, things are subtle:

sub main {
    $var = 42;
}

&main;

print "var: $var\n";

The output is var: 42 , $var is a global variable even it's defined in a function block! Actually in Perl, any variable is declared as global by default.

The lesson is to always add use strict; use warnings; at the beginning of a Perl program, which will force the programmer to declare the lexical variable explicitly, so that we don't get messed up by some mistakes taken for granted.

Ólafur Waage ,May 10, 2009 at 10:25

The perldoc has a good definition of our.

Unlike my, which both allocates storage for a variable and associates a simple name with that storage for use within the current scope, our associates a simple name with a package variable in the current package, for use within the current scope. In other words, our has the same scoping rules as my, but does not necessarily create a variable.

Cosmicnet ,Nov 22, 2014 at 13:57

This is only somewhat related to the question, but I've just discovered a (to me) obscure bit of perl syntax that you can use with "our" (package) variables that you can't use with "my" (local) variables.
#!/usr/bin/perl

our $foo = "BAR";

print $foo . "\n";
${"foo"} = "BAZ";
print $foo . "\n";

Output:

BAR
BAZ

This won't work if you change 'our' to 'my'.

Okuma.Scott ,Sep 6, 2014 at 20:13

print "package is: " . __PACKAGE__ . "\n";
our $test = 1;
print "trying to print global var from main package: $test\n";

package Changed;

{
        my $test = 10;
        my $test1 = 11;
        print "trying to print local vars from a closed block: $test, $test1\n";
}

&Check_global;

sub Check_global {
        print "trying to print global var from a function: $test\n";
}
print "package is: " . __PACKAGE__ . "\n";
print "trying to print global var outside the func and from \"Changed\" package:     $test\n";
print "trying to print local var outside the block $test1\n";

Will Output this:

package is: main
trying to print global var from main package: 1
trying to print local vars from a closed block: 10, 11
trying to print global var from a function: 1
package is: Changed
trying to print global var outside the func and from "Changed" package: 1
trying to print local var outside the block

In case using "use strict" will get this failure while attempting to run the script:

Global symbol "$test1" requires explicit package name at ./check_global.pl line 24.
Execution of ./check_global.pl aborted due to compilation errors.

Nathan Fellman ,Nov 5, 2015 at 14:03

Just try to use the following program :
#!/usr/local/bin/perl
use feature ':5.10';
#use warnings;
package a;
{
my $b = 100;
our $a = 10;


print "$a \n";
print "$b \n";
}

package b;

#my $b = 200;
#our $a = 20 ;

print "in package b value of  my b $a::b \n";
print "in package b value of our a  $a::a \n";

Nathan Fellman ,May 16, 2013 at 11:07

#!/usr/bin/perl -l

use strict;

# if string below commented out, prints 'lol' , if the string enabled, prints 'eeeeeeeee'
#my $lol = 'eeeeeeeeeee' ;
# no errors or warnings at any case, despite of 'strict'

our $lol = eval {$lol} || 'lol' ;

print $lol;

Evgeniy ,Jan 27, 2016 at 4:57

Let us think what an interpreter actually is: it's a piece of code that stores values in memory and lets the instructions in a program that it interprets access those values by their names, which are specified inside these instructions. So, the big job of an interpreter is to shape the rules of how we should use the names in those instructions to access the values that the interpreter stores.

On encountering "my", the interpreter creates a lexical variable: a named value that the interpreter can access only while it executes a block, and only from within that syntactic block. On encountering "our", the interpreter makes a lexical alias of a package variable: it binds a name, which the interpreter is supposed from then on to process as a lexical variable's name, until the block is finished, to the value of the package variable with the same name.

The effect is that you can then pretend that you're using a lexical variable and bypass the rules of 'use strict' on full qualification of package variables. Since the interpreter automatically creates package variables when they are first used, the side effect of using "our" may also be that the interpreter creates a package variable as well. In this case, two things are created: a package variable, which the interpreter can access from everywhere, provided it's properly designated as requested by 'use strict' (prepended with the name of its package and two colons), and its lexical alias.

Sources:

[Oct 09, 2019] Perl Import Package in different Namespace

Oct 09, 2019 | stackoverflow.com

Perl Import Package in different Namespace Ask Question Asked 1 year ago Active 7 months ago Viewed 150 times We're doing things differently. View all 8 job openings! 2


choroba ,Sep 28, 2018 at 22:17

is it possible to import ( use ) a perl module within a different namespace?

Let's say I have a Module A (XS Module with no methods Exported @EXPORT is empty) and I have no way of changing the module.

This Module has a Method A::open

currently I can use that Module in my main program (package main) by calling A::open I would like to have that module inside my package main so that I can directly call open

I tried to manually push every key of %A:: into %main:: however that did not work as expected.

The only way that I know to achieve what I want is by using package A; inside my main program, effectively changing the package of my program from main to A . Im not satisfied with this. I would really like to keep my program inside package main.

Is there any way to achieve this and still keep my program in package main?

Offtopic: Yes I know usually you would not want to import everything into your namespace but this module is used by us extensively and we don't want to type A:: (well the actual module name is way longer which isn't making the situation better)in front of hundreds or thousands of calls

Grinnz ,Oct 1, 2018 at 6:26

This is one of those "impossible" situations, where the clear solution -- to rework that module -- is off limits.

But, you can alias that package's subs names, from its symbol table, to the same names in main . Worse than being rude, this comes with a glitch: it catches all names that that package itself imported in any way. However, since this package is a fixed quantity it stands to reason that you can establish that list (and even hard-code it). It is just this one time, right?

main

use warnings;
use strict;
use feature 'say';

use OffLimits;

GET_SUBS: {
    # The list of names to be excluded
    my $re_exclude = qr/^(?:BEGIN|import)$/;  # ...
    my @subs = grep { !/$re_exclude/ } sort keys %OffLimits::;
    no strict 'refs';
    for my $sub_name (@subs) {
        *{ $sub_name } = \&{ 'OffLimits::' . $sub_name };
    }   
};

my $name = name('name() called from ' . __PACKAGE__);
my $id   = id('id() called from ' . __PACKAGE__);

say "name() returned: $name";
say "id()   returned: $id";

with OffLimits.pm

package OffLimits;    
use warnings;
use strict;

sub name { return "In " .  __PACKAGE__ . ": @_" }
sub id   { return "In " .  __PACKAGE__ . ": @_" }

1;

It prints

name() returned: In OffLimits: name() called from  main
id()   returned: In OffLimits: id() called from  main

You may need that code in a BEGIN block, depending on other details.

Another option is of course to hard-code the subs to be "exported" (in @subs ). Given that the module is in practice immutable this option is reasonable and more reliable.


This can also be wrapped in a module, so that you have the normal, selective, importing.

WrapOffLimits.pm

package WrapOffLimits;
use warnings;
use strict;

use OffLimits;

use Exporter qw(import);

our @sub_names;
our @EXPORT_OK   = @sub_names;
our %EXPORT_TAGS = (all => \@sub_names);

BEGIN { 
    # Or supply a hard-coded list of all module's subs in @sub_names
    my $re_exclude = qr/^(?:BEGIN|import)$/;  # ...
    @sub_names = grep { !/$re_exclude/ } sort keys %OffLimits::;

    no strict 'refs';
    for my $sub_name (@sub_names) {
        *{ $sub_name } = \&{ 'OffLimits::' . $sub_name };
    }   
};
1;

and now in the caller you can import either only some subs

use WrapOffLimits qw(name);

or all

use WrapOffLimits qw(:all);

with otherwise the same main as above for a test.

The module name is hard-coded, which should be OK as this is meant only for that module.


The following is added mostly for completeness.

One can pass the module name to the wrapper by writing one's own import sub, which is what gets used then. The import list can be passed as well, at the expense of an awkward interface of the use statement.

It goes along the lines of

package WrapModule;
use warnings;
use strict;

use OffLimits;

use Exporter qw();  # will need our own import 

our ($mod_name, @sub_names);

our @EXPORT_OK   = @sub_names;
our %EXPORT_TAGS = (all => \@sub_names);

sub import {
    my $mod_name = splice @_, 1, 1;  # remove mod name from @_ for goto

    my $re_exclude = qr/^(?:BEGIN|import)$/;  # etc

    no strict 'refs';
    @sub_names = grep { !/$re_exclude/ } sort keys %{ $mod_name . '::'};    
    for my $sub_name (@sub_names) {    
        *{ $sub_name } = \&{ $mod_name . '::' . $sub_name };
    }   

    push @EXPORT_OK, @sub_names;

    goto &Exporter::import;
}
1;

what can be used as

use WrapModule qw(OffLimits name id);  # or (OffLimits :all)

or, with the list broken-up so to remind the user of the unusual interface

use WrapModule 'OffLimits', qw(name id);

When used with the main above this prints the same output.

The use statement ends up using the import sub defined in the module, which exports symbols by writing to the caller's symbol table. (If no import sub is written then the Exporter 's import method is nicely used, which is how this is normally done.)

This way we are able to unpack the arguments and have the module name supplied at use invocation. With the import list supplied as well now we have to push manually to @EXPORT_OK since this can't be in the BEGIN phase. In the end the sub is replaced by Exporter::import via the (good form of) goto , to complete the job.

Simerax ,Sep 30, 2018 at 10:19

You can forcibly "import" a function into main using glob assignment to alias the subroutine (and you want to do it in BEGIN so it happens at compile time, before calls to that subroutine are parsed later in the file):
use strict;
use warnings;
use Other::Module;

BEGIN { *open = \&Other::Module::open }

However, another problem you might have here is that open is a builtin function, which may cause some problems . You can add use subs 'open'; to indicate that you want to override the built-in function in this case, since you aren't using an actual import function to do so.

Grinnz ,Sep 30, 2018 at 17:33

Here is what I now came up with. Yes this is hacky and yes I also feel like I opened pandoras box with this. However at least a small dummy program ran perfectly fine.

I renamed the module in my code again. In my original post I used the example A::open actually this module does not contain any method/variable reserved by the perl core. This is why I blindly import everything here.

BEGIN {
    # using the caller to determine the parent. Usually this is main but maybe we want it somewhere else in some cases
    my ($parent_package) = caller;

    package A;

    foreach (keys(%A::)) {
        if (defined $$_) {
            eval '*'.$parent_package.'::'.$_.' = \$A::'.$_;
        }
        elsif (%$_) {
            eval '*'.$parent_package.'::'.$_.' = \%A::'.$_;
        }
        elsif (@$_) {
            eval '*'.$parent_package.'::'.$_.' = \@A::'.$_;
        }
        else {
            eval '*'.$parent_package.'::'.$_.' = \&A::'.$_;
        }
    }
}

[Oct 09, 2019] oop - Perl Importing Variables From Calling Module

Oct 09, 2019 | stackoverflow.com

Perl Importing Variables From Calling Module Ask Question Asked 9 years, 1 month ago Active 9 years, 1 month ago Viewed 4k times 0 1


Russell C. ,Aug 31, 2010 at 20:31

I have a Perl module (Module.pm) that initializes a number of variables, some of which I'd like to import ($VAR2, $VAR3) into additional submodules that it might load during execution.

The way I'm currently setting up Module.pm is as follows:

package Module;

use warnings;
use strict;

use vars qw($SUBMODULES $VAR1 $VAR2 $VAR3);

require Exporter;
our @ISA = qw(Exporter);
our @EXPORT = qw($VAR2 $VAR3);

sub new {
    my ($package) = @_;
    my $self = {};
    bless ($self, $package);
    return $self;
}

sub SubModules1 {
    my $self = shift;
    if($SUBMODULES->{'1'}) { return $SUBMODULES->{'1'}; }

    # Load & cache submodule
    require Module::SubModule1;
    $SUBMODULES->{'1'} = Module::SubModule1->new(@_);    
    return $SUBMODULES->{'1'};
}

sub SubModules2 {
    my $self = shift;
    if($SUBMODULES->{'2'}) { return $SUBMODULES->{'2'}; }

    # Load & cache submodule
    require Module::SubModule2;
    $SUBMODULES->{'2'} = Module::SubModule2->new(@_);    
    return $SUBMODULES->{'2'};
}

Each submodule is structured as follows:

package Module::SubModule1;

use warnings;
use strict;
use Carp;

use vars qw();

sub new {
    my ($package) = @_;
    my $self = {};
    bless ($self, $package);
    return $self;
}

I want to be able to import the $VAR2 and $VAR3 variables into each of the submodules without having to reference them as $Module::VAR2 and $Module::VAR3. I noticed that the calling script is able to access both the variables that I have exported in Module.pm in the desired fashion but SubModule1.pm and SubModule2.pm still have to reference the variables as being from Module.pm.

I tried updating each submodule as follows which unfortunately didn't work I was hoping:

package Module::SubModule1;

use warnings;
use strict;
use Carp;

use vars qw($VAR2 $VAR3);

sub new {
    my ($package) = @_;
    my $self = {};
    bless ($self, $package);
    $VAR2 = $Module::VAR2;
    $VAR3 = $Module::VAR3;
    return $self;
}

Please let me know how I can successfully export $VAR2 and $VAR3 from Module.pm into each Submodule. Thanks in advance for your help!

Russell C. ,Aug 31, 2010 at 22:37

In your submodules, are you forgetting to say
use Module;

? Calling use Module from another package (say Module::Submodule9 ) will try to run the Module::import method. Since you don't have that method, it will call the Exporter::import method, and that is where the magic that exports Module 's variables into the Module::Submodule9 namespace will happen.


In your program there is only one Module namespace and only one instance of the (global) variable $Module::VAR2 . Exporting creates aliases to this variable in other namespaces, so the same variable can be accessed in different ways. Try this in a separate script:

package Whatever;
use Module;
use strict;
use vars qw($VAR2);

$Module::VAR2 = 5;
print $Whatever::VAR2;    # should be 5.
$VAR2 = 14;               # same as $Whatever::VAR2 = 14
print $Module::VAR2;      # should be 14

Russell C. ,Aug 31, 2010 at 21:38

Well there is the easy way:

In M.pm:

package M;

use strict;
use warnings;

#our is better than "use vars" for creating package variables
#it creates an alias to $M::foo named $foo in the current lexical scope 
our $foo = 5;

sub inM { print "$foo\n" }

1;

In M/S.pm

package M;

#creates an alias to $M::foo that will last for the entire scope,
#in this case the entire file
our $foo;

package M::S;

use strict;
use warnings;

sub inMS { print "$foo\n" }

1;

In the script:

#!/usr/bin/perl

use strict;
use warnings;

use M;
use M::S;

M::inM();
M::S::inMS();

But I would advise against this. Global variables are not a good practice, and sharing global variables between modules is even worse.

[Oct 09, 2019] gzip - How can I recover files from a corrupted .tar.gz archive - Stack Overflow

Oct 09, 2019 | stackoverflow.com

15


George ,Jun 24, 2016 at 2:49

Are you sure that it is a gzip file? I would first run 'file SMS.tar.gz' to validate that.

Then I would read the The gzip Recovery Toolkit page.

JohnEye ,Oct 4, 2016 at 11:27

Recovery is possible but it depends on what caused the corruption.

If the file is just truncated, getting some partial result out is not too hard; just run

gunzip < SMS.tar.gz > SMS.tar.partial

which will give some output despite the error at the end.

If the compressed file has large missing blocks, it's basically hopeless after the bad block.

If the compressed file is systematically corrupted in small ways (e.g. transferring the binary file in ASCII mode, which smashes carriage returns and newlines throughout the file), it is possible to recover but requires quite a bit of custom programming, it's really only worth it if you have absolutely no other recourse (no backups) and the data is worth a lot of effort. (I have done it successfully.) I mentioned this scenario in a previous question .

The answers for .zip files differ somewhat, since zip archives have multiple separately-compressed members, so there's more hope (though most commercial tools are rather bogus, they eliminate warnings by patching CRCs, not by recovering good data). But your question was about a .tar.gz file, which is an archive with one big member.

,

Here is one possible scenario that we encountered. We had a tar.gz file that would not decompress, trying to unzip gave the error:
gzip -d A.tar.gz
gzip: A.tar.gz: invalid compressed data--format violated

I figured out that the file may been originally uploaded over a non binary ftp connection (we don't know for sure).

The solution was relatively simple using the unix dos2unix utility

dos2unix A.tar.gz
dos2unix: converting file A.tar.gz to UNIX format ...
tar -xvf A.tar
file1.txt
file2.txt 
....etc.

It worked! This is one slim possibility, and maybe worth a try - it may help somebody out there.

[Oct 08, 2019] Perl constant array

Oct 08, 2019 | stackoverflow.com

Ask Question Asked 6 years, 1 month ago Active 4 years ago Viewed 5k times 4 1


Alec ,Sep 5, 2018 at 8:25

use constant {
    COLUMNS => qw/ TEST1 TEST2 TEST3 /,
}

Can I store an array using the constant package in Perl?

Whenever I go on to try to use the array like my @attr = (COLUMNS); , it does not contain the values.

Сухой27 ,Aug 12, 2013 at 13:37

use constant {
  COLUMNS => [qw/ TEST1 TEST2 TEST3 /],
};

print @{+COLUMNS};

> ,

Or remove the curly braces as the docs show :-
  1 use strict;
  2 use constant COLUMNS => qw/ TEST1 TEST2 TEST3 /;
  3 
  4 my @attr = (COLUMNS);
  5 print @attr;

which gives :-

 % perl test.pl
TEST1TEST2TEST3

Your code actually defines two constants COLUMNS and TEST2 :-

use strict;
use constant { COLUMNS => qw/ TEST1 TEST2 TEST3 /, };

my @attr = (COLUMNS);
print @attr;
print TEST2

and gives :-

% perl test.pl
TEST1TEST3

[Oct 07, 2019] How to commit to remote git repository

Apr 28, 2012 | stackoverflow.com

Ahmed ,Apr 28, 2012 at 14:32

I am new to git.
I have done a clone of remote repo as follows
git clone https://myusername@something.com/repo.git

then I did

git checkout master

made some changes and committed these changes to my local repository like below..

git add .

git commit -m "my changes"

Now I have to push these changes to the remote repository. I am not sure what to do.

Would I do a merge of my repo to remote ? what steps do I need to take ?

I have git bash and git gui

please advise,
thanks,

zafarkhaja ,Apr 28, 2012 at 14:39

All You have to do is git push origin master , where origin is the default name (alias) of Your remote repository and master is the remote branch You want to push Your changes to.

You may also want to check these out:

  1. http://gitimmersion.com/
  2. http://progit.org/book/

Sergey K. ,Apr 28, 2012 at 14:54

You just need to make sure you have the rights to push to the remote repository and do
git push origin master

or simply

git push

haziz ,Apr 28, 2012 at 21:30

git push

or

git push server_name master

should do the trick, after you have made a commit to your local repository.

Bryan Oakley ,Apr 28, 2012 at 14:34

Have you tried git push ? gitref.org has a nice section dealing with remote repositories .

You can also get help from the command line using the --help option. For example:

% git push --help
GIT-PUSH(1)                             Git Manual                             GIT-PUSH(1)



NAME
       git-push - Update remote refs along with associated objects

SYNOPSIS
       git push [--all | --mirror | --tags] [-n | --dry-run] [--receive-pack=<git-receive-pack>]
                  [--repo=<repository>] [-f | --force] [-v | --verbose] [-u | --set-upstream]
                  [<repository> [<refspec>...]]
...

[Sep 30, 2019] Get command line arguments as string

Sep 30, 2019 | stackoverflow.com

KocT9H ,Jun 6, 2016 at 12:58

I want to print all command line arguments as a single string. Example of how I call my script and what I expect to be printed:
./RunT.py mytst.tst -c qwerty.c

mytst.tst -c qwerty.c

The code that does that:

args = str(sys.argv[1:])
args = args.replace("[","")
args = args.replace("]","")
args = args.replace(",","")
args = args.replace("'","")
print args

I did all replaces because sys.argv[1:] returns this:

['mytst.tst', '-c', 'qwerty.c']

Is there a better way to get same result? I don't like those multiple replace calls

KocT9H ,Apr 5 at 12:16

An option:
import sys
' '.join(sys.argv[1:])

The join() function joins its arguments by whatever string you call it on. So ' '.join(...) joins the arguments with single spaces ( ' ' ) between them.

cxw ,Apr 5 at 12:18

The command line arguments are already handled by the shell before they are sent into sys.argv . Therefore, shell quoting and whitespace are gone and cannot be exactly reconstructed.

Assuming the user double-quotes strings with spaces, here's a python program to reconstruct the command string with those quotes.

commandstring = '';  

for arg in sys.argv[1:]:          # skip sys.argv[0] since the question didn't ask for it
    if ' ' in arg:
        commandstring+= '"{}"  '.format(arg) ;   # Put the quotes back in
    else:
        commandstring+="{}  ".format(arg) ;      # Assume no space => no quotes

print(commandstring);

For example, the command line

./saferm.py sdkf lsadkf -r sdf -f sdf -fs -s "flksjfksdkfj sdfsdaflkasdf"

will produce the same arguments as output:

sdkf lsadkf -r sdf -f sdf -fs -s "flksjfksdkfj sdfsdaflkasdf"

since the user indeed double-quoted only arguments with strings.

TryToSolveItSimple ,Apr 5 at 12:27

You're getting a list object with all of your arguments when you use the syntax [1:] which goes from the second argument to the last. You could run a for each loop to join them into one string:
args = sys.argv[1:]
result = ''

for arg in args:
    result += " " + arg

pushpen.paul ,Jul 13 at 1:39

None of the previous answers properly escape all possible arguments, like empty args or those containing quotes. The closest you can get with minimal code is to use shlex.quote (available since Python 3.3):
import shlex
cmdline = " ".join(map(shlex.quote, sys.argv[1:]))

[Sep 30, 2019] How can I convert a string to a number in Perl?

Sep 30, 2019 | stackoverflow.com

Ask Question Asked 10 years, 10 months ago Active 4 years, 7 months ago Viewed 264k times 81 7


TimK ,Mar 1, 2016 at 22:51

How would I convert a string holding a number into its numeric value in Perl?

OrangeDog ,May 22, 2014 at 15:24

You don't need to convert it at all:
% perl -e 'print "5.45" + 0.1;'
5.55

,

This is a simple solution:

Example 1

my $var1 = "123abc";
print $var1 + 0;

Result

123

Example 2

my $var2 = "abc123";
print $var2 + 0;

Result

0

[Sep 28, 2019] Python quoting conventions

Sep 28, 2019 | stackoverflow.com

Share a link to this answer Copy link edited Jul 7 '11 at 14:11 answered Sep 11 '08 at 10:06 Will Harris Will Harris 20.4k 11 11 gold badges 60 60 silver badges 63 63 bronze badges

| show 2 more comments 96 votes

mlissner ,Aug 6, 2011 at 6:19

Quoting the official docs at https://docs.python.org/2.0/ref/strings.html :

In plain English: String literals can be enclosed in matching single quotes (') or double quotes (").

So there is no difference. Instead, people will tell you to choose whichever style that matches the context, and to be consistent . And I would agree - adding that it is pointless to try to come up with "conventions" for this sort of thing because you'll only end up confusing any newcomers.

eksortso ,Jun 10, 2013 at 23:36

I used to prefer ' , especially for '''docstrings''' , as I find """this creates some fluff""" . Also, ' can be typed without the Shift key on my Swiss German keyboard.

I have since changed to using triple quotes for """docstrings""" , to conform to PEP 257 .

Garth Kidd ,Sep 11, 2008 at 10:21

I'm with Will:

I'll stick with that even if it means a lot of escaping.

I get the most value out of single quoted identifiers standing out because of the quotes. The rest of the practices are there just to give those single quoted identifiers some standing room.

Tony Meyer ,Sep 11, 2008 at 8:33

If the string you have contains one, then you should use the other. For example, "You're able to do this" , or 'He said "Hi!"' . Other than that, you should simply be as consistent as you can (within a module, within a package, within a project, within an organisation).

If your code is going to be read by people who work with C/C++ (or if you switch between those languages and Python), then using '' for single-character strings, and "" for longer strings might help ease the transition. (Likewise for following other languages where they are not interchangeable).

The Python code I've seen in the wild tends to favour " over ' , but only slightly. The one exception is that """these""" are much more common than '''these''' , from what I have seen.

jblocksom ,Jul 30, 2009 at 20:35

Triple quoted comments are an interesting subtopic of this question. PEP 257 specifies triple quotes for doc strings . I did a quick check using Google Code Search and found that triple double quotes in Python are about 10x as popular as triple single quotes -- 1.3M vs 131K occurrences in the code Google indexes. So in the multi line case your code is probably going to be more familiar to people if it uses triple double quotes.

Paolo ,Sep 30, 2013 at 15:39

"If you're going to use apostrophes, 
       ^

you'll definitely want to use double quotes".
   ^

For that simple reason, I always use double quotes on the outside. Always

Speaking of fluff, what good is streamlining your string literals with ' if you're going to have to use escape characters to represent apostrophes? Does it offend coders to read novels? I can't imagine how painful high school English class was for you!

dolma33 ,Mar 17, 2014 at 23:13

Python uses quotes something like this:
mystringliteral1="this is a string with 'quotes'"
mystringliteral2='this is a string with "quotes"'
mystringliteral3="""this is a string with "quotes" and more 'quotes'"""
mystringliteral4='''this is a string with 'quotes' and more "quotes"'''
mystringliteral5='this is a string with \"quotes\"'
mystringliteral6='this is a string with \042quotes\042'
mystringliteral6='this is a string with \047quotes\047'

print mystringliteral1
print mystringliteral2
print mystringliteral3
print mystringliteral4
print mystringliteral5
print mystringliteral6

Which gives the following output:

this is a string with 'quotes'
this is a string with "quotes"
this is a string with "quotes" and more 'quotes'
this is a string with 'quotes' and more "quotes"
this is a string with "quotes"
this is a string with 'quotes'

Matt Sheppard ,Sep 11, 2008 at 8:40

I use double quotes in general, but not for any specific reason - Probably just out of habit from Java.

I guess you're also more likely to want apostrophes in an inline literal string than you are to want double quotes.

schwa ,Sep 28, 2008 at 3:35

Personally I stick with one or the other. It doesn't matter. And providing your own meaning to either quote is just to confuse other people when you collaborate.

maxpolk ,Sep 21, 2013 at 17:45

It's probably a stylistic preference more than anything. I just checked PEP 8 and didn't see any mention of single versus double quotes.

I prefer single quotes because its only one keystroke instead of two. That is, I don't have to mash the shift key to make single quote.

stivlo ,Mar 11, 2012 at 4:25

In Perl you want to use single quotes when you have a string which doesn't need to interpolate variables or escaped characters like \n, \t, \r, etc.

PHP makes the same distinction as Perl: content in single quotes will not be interpreted (not even \n will be converted), as opposed to double quotes which can contain variables to have their value printed out.

Python does not, I'm afraid. Technically seen, there is no $ token (or the like) to separate a name/text from a variable in Python. Both features make Python more readable, less confusing, after all. Single and double quotes can be used interchangeably in Python.

Andrew Dalke ,Mar 10, 2009 at 6:25

I chose to use double quotes because they are easier to see.

Alphy

add a comment ,Jul 22, 2009 at 19:50
I just use whatever strikes my fancy at the time; it's convenient to be able to switch between the two at a whim!

Of course, when quoting quote characetrs, switching between the two might not be so whimsical after all...

Vinko Vrsalovic ,Sep 11, 2008 at 8:25

Your team's taste or your project's coding guidelines.

If you are in a multilanguage environment, you might wish to encourage the use of the same type of quotes for strings that the other language uses, for instance. Else, I personally like best the look of '

Mario F ,Sep 11, 2008 at 8:28

None as far as I know. Although if you look at some code, " " is commonly used for strings of text (I guess ' is more common inside text than "), and ' ' appears in hashkeys and things like that.

Acumenus ,Apr 16, 2013 at 22:32

I aim to minimize both pixels and surprise. I typically prefer ' in order to minimize pixels, but " instead if the string has an apostrophe, again to minimize pixels. For a docstring, however, I prefer """ over ''' because the latter is non-standard, uncommon, and therefore surprising. If now I have a bunch of strings where I used " per the above logic, but also one that can get away with a ' , I may still use " in it to preserve consistency, only to minimize surprise.

Perhaps it helps to think of the pixel minimization philosophy in the following way. Would you rather that English characters looked like A B C or AA BB CC ? The latter choice wastes 50% of the non-empty pixels.

Philipp ,Jul 5, 2010 at 13:06

I use double quotes because I have been doing so for years in most languages (C++, Java, VB ) except Bash, because I also use double quotes in normal text and because I'm using a (modified) non-English keyboard where both characters require the shift key.

Adam Smith ,Dec 16, 2013 at 23:38

' = "

/ = \ = \\

example :

f = open('c:\word.txt', 'r')
f = open("c:\word.txt", "r")
f = open("c:/word.txt", "r")
f = open("c:\\\word.txt", "r")

Results are the same

=>> no, they're not the same. A single backslash will escape characters. You just happen to luck out in that example because \k and \w aren't valid escapes like \t or \n or \\ or \"

If you want to use single backslashes (and have them interpreted as such), then you need to use a "raw" string. You can do this by putting an ' r ' in front of the string

im_raw = r'c:\temp.txt'
non_raw = 'c:\\temp.txt'
another_way = 'c:/temp.txt'

As far as paths in Windows are concerned, forward slashes are interpreted the same way. Clearly the string itself is different though. I wouldn't guarantee that they're handled this way on an external device though.

[Sep 28, 2019] python - Get command line arguments as string - Stack Overflow

Sep 28, 2019 | stackoverflow.com

Get command line arguments as string Ask Question Asked 3 years, 3 months ago Active 2 months ago Viewed 21k times 7 1


KocT9H ,Jun 6, 2016 at 12:58

I want to print all command line arguments as a single string. Example of how I call my script and what I expect to be printed:
./RunT.py mytst.tst -c qwerty.c

mytst.tst -c qwerty.c

The code that does that:

args = str(sys.argv[1:])
args = args.replace("[","")
args = args.replace("]","")
args = args.replace(",","")
args = args.replace("'","")
print args

I did all replaces because sys.argv[1:] returns this:

['mytst.tst', '-c', 'qwerty.c']

Is there a better way to get same result? I don't like those multiple replace calls

KocT9H ,Apr 5 at 12:16

An option:
import sys
' '.join(sys.argv[1:])

The join() function joins its arguments by whatever string you call it on. So ' '.join(...) joins the arguments with single spaces ( ' ' ) between them.

cxw ,Apr 5 at 12:18

The command line arguments are already handled by the shell before they are sent into sys.argv . Therefore, shell quoting and whitespace are gone and cannot be exactly reconstructed.

Assuming the user double-quotes strings with spaces, here's a python program to reconstruct the command string with those quotes.

commandstring = '';  

for arg in sys.argv[1:]:          # skip sys.argv[0] since the question didn't ask for it
    if ' ' in arg:
        commandstring+= '"{}"  '.format(arg) ;   # Put the quotes back in
    else:
        commandstring+="{}  ".format(arg) ;      # Assume no space => no quotes

print(commandstring);

For example, the command line

./saferm.py sdkf lsadkf -r sdf -f sdf -fs -s "flksjfksdkfj sdfsdaflkasdf"

will produce the same arguments as output:

sdkf lsadkf -r sdf -f sdf -fs -s "flksjfksdkfj sdfsdaflkasdf"

since the user indeed double-quoted only arguments with strings.

TryToSolveItSimple ,Apr 5 at 12:27

You're getting a list object with all of your arguments when you use the syntax [1:] which goes from the second argument to the last. You could run a for each loop to join them into one string:
args = sys.argv[1:]
result = ''

for arg in args:
    result += " " + arg

pushpen.paul ,Jul 13 at 1:39

None of the previous answers properly escape all possible arguments, like empty args or those containing quotes. The closest you can get with minimal code is to use shlex.quote (available since Python 3.3):
import shlex
cmdline = " ".join(map(shlex.quote, sys.argv[1:]))

> ,

Or can do format , and multiply the string by length of sys.argv :
(' {}'*len(sys.argv)).lstrip().format(*sys.argv)

Or can do % :

(' %s'*len(sys.argv)).lstrip()%sys.argv

[Sep 24, 2019] How to properly use the try catch in perl that error.pm provides?

Sep 24, 2019 | stackoverflow.com

Sinan Ünür ,Apr 28, 2012 at 18:07

Last I checked, Error was deprecated. But here's how you would do it without that module:
eval {
    die "Oops!";
    1;
} or do {
    my $e = $@;
    print("Something went wrong: $e\n");
};

Basically, use eval instead of try , die instead of throw , and look for the exception in $@ . The true value at the end of the eval block is part of an idiom to prevent $@ from unintentionally changing before it is used again in Perl versions older than 5.14, see P::C::P::ErrorHandling::RequireCheckingReturnValueOfEval for details. For example, this code suffers from this flaw.

# BAD, DO NOT USE WITH PERLS OLDER THAN 5.14
eval {
    die "Oops!";
};
if (my $e = $@) {
    print("Something went wrong: $e\n");
}
# BAD, DO NOT USE WITH PERLS OLDER THAN 5.14

But note that many Perl operations do not raise exceptions when they fail; they simply return an error code. This behavior can be altered via autodie for builtins and standard modules. If you're using autodie , then the standard way of doing try/catch is this (straight out of the autodie perldoc):

use feature qw(switch);

eval {
   use autodie;

   open(my $fh, '<', $some_file);

   my @records = <$fh>;

   # Do things with @records...

   close($fh);

};

given ($@) {
   when (undef)   { say "No error";                    }
   when ('open')  { say "Error from open";             }
   when (':io')   { say "Non-open, IO error.";         }
   when (':all')  { say "All other autodie errors."    }
   default        { say "Not an autodie error at all." }
}

For getting a stacktrace, look at Carp .

> ,

If you want something a bit more powerful than Try::Tiny, you might want to try looking at the TryCatch module in CPAN.

[Sep 24, 2019] What is the best way to handle exceptions in Perl - Stack Overflow

Sep 24, 2019 | stackoverflow.com

Michael Carman ,Apr 8 at 9:52

The consensus of the Perl community seems to be that Try::Tiny is the preferred way of doing exception handling. The "lenient policy" you refer to is probably due to a combination of:

Note that the last item means that you'll see a lot of code like this:

eval { something() };
if ($@) {
    warn "Oh no! [$@]\n";
}

That's exception handling even though it doesn't use try/catch syntax. It's fragile, though, and will break in a number of subtle edge cases that most people don't think about.

Try::Tiny and the other exception handling modules on CPAN were written to make it easier to get right.

1. C does have setjmp() and longjmp() , which can be used for a very crude form of exception handling.

,

Never test $@ as is, because it is a global variable, so even the test itself can change it.

General eval-template:

my $result;

eval {
    $result= something();
    # ...
    1;  # ok
} or do {
    my $eval_error= $@ || "error";
    # ...
    die $eval_error;
};  # needs a semicolon

In practice that is the lightest way. It still leaves a tiny room for funny $@ behaviour, but nothing that really concerned me enough.

[Sep 21, 2019] How Did Perl Lose Ground to Bash?

Notable quotes:
"... It baffles me the most because the common objection to Perl is legibility. Even if you assume that the objection is made from ignorance - i.e. not even having looked at some Perl to gauge its legibility - the nonsense you see in a complex bash script is orders of magnitude worse! ..."
"... Maybe it's not reassuring to hear that, but I took an interest in Perl precisely because it's seen as an underdog and "dead" despite having experienced users and a lot of code, kind of like TCL, Prolog, or Ada. ..."
"... There's a long history of bad code written by mediocre developers who became the only one who could maintain the codebase until they no longer worked for the organization. The next poor sap to go in found a mess of a codebase and did their best to not break it further. After a few iterations, the whole thing is ready for /dev/null and Perl gets the blame. ..."
"... All in all, Perl is still my first go-to language, but there are definitely some things I wish it did better. ..."
"... The Perl leadership Osborned itself with Perl6. 20/20 hindsight says the new project should have been given a different name at conception, that way all the "watch this space -- under construction" signage wouldn't have steered people away from perfectly usable Perl5. Again, IMO. ..."
"... I don't observe the premise at all though. Is bash really gaining ground over anything recently? ..."
"... Python again is loved, because "taught by rote" idiots. Now you can give them pretty little packages. And it's no wonder they can do little better than be glorified system admins (which id rather have a real sys admin, since he's likely to understand Perl) ..."
"... Making a new language means lots of new training. Lots of profit in this. Nobody profits from writing new books on old languages. Lots of profit in general from supporting a new language. In the end, owning the language gets you profits. ..."
"... And I still don't get why tab for blocks python is even remotely more readable than Perl. ..."
"... If anything, JavaScript is pretty dang godly at what it does, I understand why that's popular. But I don't get python one bit, except to employ millions of entry level minions who can't think on their own. ..."
"... "Every teacher I know has students using it. We do it because it's an easy language, there's only one way to do it, and with whitespace as syntax it's easy to grade. We don't teach it because it is some powerful or exceptional language. " ..."
Sep 21, 2019 | www.reddit.com

How Did Perl Lose Ground to Bash?

Setting aside Perl vs. Python for the moment, how did Perl lose ground to Bash? It used to be that Bash scripts often got replaced by Perl scripts because Perl was more powerful. Even with very modern versions of Bash, Perl is much more powerful.

The Linux Standards Base (LSB) has helped ensure that certain tools are in predictable locations. Bash has gotten a bit more powerful since the release of 4.x, sure. Arrays, handicapped to 2-D arrays, have improved somewhat. There is a native regex engine in Bash 3.x, which admit is a big deal. There is also support for hash maps.

This is all good stuff for Bash. But, none of this is sufficient to explain why Perl isn't the thing you learn after Bash, or, after Bash and Python; take your pick. Thoughts?

28 comments 75% Upvoted What are your thoughts? Log in or Sign up log in sign up Sort by

oldmanwillow21 9 points · 9 days ago

Because Perl has suffered immensely in the popularity arena and is now viewed as undesirable. It's not that Bash is seen as an adequate replacement for Perl, that's where Python has landed.

emilper 8 points · 8 days ago

How did Perl5 lose ground to anything else?

Thusly

- "thou must use Moose for everything" -> "Perl is too slow" -> rewrite in Python because the architect loves Python -> Python is even slower -> architect shunned by the team and everything new written in Go, nobody dares to complain about speed now because the budget people don't trust them -> Perl is slow

- "globals are bad, singletons are good" -> spaghetti -> Perl is unreadable

- "lets use every single item from the gang of four book" -> insanity -> Perl is bad

- "we must be more OOP" -> everything is a faux object with everything else as attributes -> maintenance team quits and they all take PHP jobs, at least the PHP people know their place in the order of things and do less hype-driven-development -> Perl is not OOP enough

- "CGI is bad" -> app needs 6.54GB of RAM for one worker -> customer refuses to pay for more RAM, fires the team, picks a PHP team to do the next version -> PHP team laughs all the way to the bank, chanting "CGI is king"

recrof 2 points · 8 days ago

"CGI is bad" is real. PSGI or FCGI is much faster for web services, and if there are memory leaks, it's always possible to debug & fix them.

Grinnz 6 points · 8 days ago

CGI is fine, when it's all you need. There are many different use cases out there. Just don't use CGI.pm .

emilper 2 points · 7 days ago

memory leaks

memory leaks ... do huge monoliths count as "memory leaks" ?

Altreus 7 points · 8 days ago

It baffles me the most because the common objection to Perl is legibility. Even if you assume that the objection is made from ignorance - i.e. not even having looked at some Perl to gauge its legibility - the nonsense you see in a complex bash script is orders of magnitude worse!

Not to mention its total lack of common language features like first-class data and... Like, a compiler...

I no longer write bash scripts because it takes about 5 lines to become unmaintainable.

crashorbit 5 points · 9 days ago

Every language that reaches functional equity with Perl is perceived as better than it. Mostly because hey, at least it's not Perl.

oldmanwillow21 15 points · 9 days ago · edited 9 days ago

Jumbled mess of thoughts surely to follow.

When I discuss projects with peers and mention that I chose to develop in Perl, the responses range from passive bemusement, to scorn, to ridicule. The assumption is usually that I'm using a dead language that's crippled in functionality and uses syntax that will surely make everyone's eyes bleed to read. This is the culture everywhere from the casual hackers to the C-suite.

I've proven at work that I can write nontrivial software using Perl. I'm still asked to use Python or Go (edit: or node, ugh) for any project that'll have contributors from other teams, or to containerize apps using Docker to remove the need for Perl knowledge for end-users (no CPAN, carton, etc.). But I'll take what I can get, and now the attitude has gone from "get with the times" or "that's cute", to "ok but I don't expect everyone else to know it".

Perl has got a lot to offer, and I vastly enjoy using it over other languages I work with. I know that all the impassioned figures in the Perl community love it just the same, but the community's got some major fragmentation going on. I understand that everyone's got ideas about the future of the language, but is this really the best time to pull the community apart? I feel like if everyone was able to let go of their ego and put their heads together to bring us to a point of stability, even a place where we're not laughed at for professing our support for the language, it would be a major step in the right direction. I think we're heading to the bottom fast, otherwise.

In that spirit of togetherness, I think the language, particularly the community, needs to be made more accessible to newcomers. Not accessible to one Perl offshoot, but accessible to Perl. It needs to be decided what Perl means in today's day and age. What can it do? Why would I want to use it over another shiny language? What are the definitive places I can go to learn more? Who else will be there? How do I contribute and grow as a Perl developer? There need to be people talking about Perl in places that aren't necessarily hubs for other Perl enthusiasts. It needs to be something business decision-makers can look at and feel confident in using.

I really hope something changes. I'd be pretty sad if I had to spend the rest of my career writing whatever the trendy language of the day is. These are just observations from someone that likes writing Perl and has been watching from the sidelines.

PhloxPaniculata 2 points · 7 days ago

Maybe it's not reassuring to hear that, but I took an interest in Perl precisely because it's seen as an underdog and "dead" despite having experienced users and a lot of code, kind of like TCL, Prolog, or Ada.

Being able to read Modern Perl for free also helped a lot. I'm still lacking experience in Perl and I've yet to write anything of importance in it because I don't see an area in which it's clearly better than anything else, either because of the language, a package, or a framework, and I don't do a lot of text-munging anymore (I'm also a fan of awk so for small tasks it has the priority).

codon011 1 point · 9 days ago

Don't call it Perl. Unfortunately. Also IME multitasking in Perl5 (or the lack thereof and/or severe issues with) has been a detriment to it's standing in a "multithread all the things" world.

crashorbit 4 points · 8 days ago

So often I see people drag themselves down that "thread my app" path. Eventually realize that they are implementing a whole multi-processing operating system inside their app rather than taking advantage of the perfectly good one they are running on.

There are several perfectly good ways to do concurrency, multitasking, async IO and so on in perl. Many work well in the single node case and in the multi-node case. Anyone who tells you that multitasking systems are easy because of some implementation language choice has not made it through the whole Dunning Kruger cycle yet.

codon011 2 points · 8 days ago

Multithreading is never easy. The processors will always manage to do things in a "wrong" order unless you are very careful with your gatekeeping. However, other languages/frameworks have paradigms that make it seem easier such that those race conditions show up much later in your product lifecycle.

codon011 3 points · 9 days ago

There's a long history of bad code written by mediocre developers who became the only one who could maintain the codebase until they no longer worked for the organization. The next poor sap to go in found a mess of a codebase and did their best to not break it further. After a few iterations, the whole thing is ready for /dev/null and Perl gets the blame.

Bash has limitations, but that (usually) means fewer ways to mess it up. There's less domain knowledge to learn, (afaik) no CPAN equivalent, and fewer issues with things like "I need to upgrade this but I can't because this other thing uses this older version which is incompatible with the newer version so now we have to maintain two versions of the library and/or interpreter."

All in all, Perl is still my first go-to language, but there are definitely some things I wish it did better.

crb3 3 points · 9 days ago · edited 9 days ago

*[e:] Consider, not just core here, but CPAN pull-in as well. I had one project clobbered on a smaller-memory machine when I tried to set up a pure-Perl scp transfer -- there wasn't room enough for the full file to transfer if it was larger than about 50k, what with all the CPAN. Shelling to commandline scp worked just fine.

beermad 2 points · 8 days ago

To be fair, wrapping a Perl script around something that's (if I read your comment right) just running SCP is adding a pointless extra layer of complexity anyway.

It's a matter of using the best tool for each particular job, not just sticking with one. My own ~/bin directory has a big mix of Perl and pure shell, depending on the complexity of the job to be done.

crb3 2 points · 8 days ago · edited 7 days ago

Agreed; I brought that example up to illustrate the bulk issue. In it, I was feeling my way, not sure how much finagling I might have to do for the task (backdoor-passing legitimate sparse but possibly quite bulky email from one server to another), which is why I initially went for the pure-Perl approach, so I'd have the mechanics exposed for any needed hackery. The experience taught me to get by more on shelling to precompiled tooling where appropriate... and a healthy respect for CPAN pull-in, [e:] the way that this module depends on that module so it gets pulled in along with its dependencies in turn, and the pileup grows in memory. There was a time or two here and there where I only needed a teeny bit of what a module does, so I went in and studied the code, then implemented it internally as a function without the object's generalities and bulk. The caution learned on ancient x86 boxes now seems appropriate on ARM boards like rPi; what goes around comes around.

minimim 1 point · 4 days ago

wouldn't have steered people away from perfectly usable Perl5

Perl5 development was completely stalled at the time. Perl6 brought not only new blood into it's own effort, it reinvigorated Perl5 in the process.

It's completely backwards to suggest Perl 5 was fine until perl6 came along. It was almost dormant and became a lively language after Perl 6 was announced.

perlancar 2 points · 8 days ago

I don't observe the premise at all though. Is bash really gaining ground over anything recently? l

linearblade 3 points · 8 days ago

Perl is better than pretty much everything g out there at what it does.

But keep in mind,

They say C sharp is loved by everyone, when in reality it's Microsoft pushing their narrative and the army of "learn by rote" engineers In developing countries

Python again is loved, because "taught by rote" idiots. Now you can give them pretty little packages. And it's no wonder they can do little better than be glorified system admins (which id rather have a real sys admin, since he's likely to understand Perl)

Making a new language means lots of new training. Lots of profit in this. Nobody profits from writing new books on old languages. Lots of profit in general from supporting a new language. In the end, owning the language gets you profits.

And I still don't get why tab for blocks python is even remotely more readable than Perl.

If anything, JavaScript is pretty dang godly at what it does, I understand why that's popular. But I don't get python one bit, except to employ millions of entry level minions who can't think on their own.

duo-rotae 6 points · 8 days ago

I know a comp sci professor. I asked why he thought Python was so popular.

"Every teacher I know has students using it. We do it because it's an easy language, there's only one way to do it, and with whitespace as syntax it's easy to grade. We don't teach it because it is some powerful or exceptional language. "

Then he said if he really needs to get something done, it's Perl or C.

linearblade 2 points · 8 days ago

Yep that's pretty much my opinion from using it.

techsnapp 1 point · 2 days ago

So is per harder than python because the lack of everyone else using it?

duo-rotae 1 point · 2 days ago

Perl has a steeper and longer learning with it. curve than Python, and there is more than one way to do anything. And there quite a few that continue coding

[Sep 19, 2019] List::MoreUtils's minmax is more efficient when you need both the min and the max (because it does fewer comparisons).

Notable quotes:
"... List::MoreUtils's minmax is more efficient when you need both the min and the max (because it does fewer comparisons). ..."
Sep 19, 2019 | stackoverflow.com

List::Util's min and max are fine,

use List::Util qw( min max );
my $min = min @numbers;
my $max = max @numbers;

But List::MoreUtils's minmax is more efficient when you need both the min and the max (because it does fewer comparisons).

use List::MoreUtils qw( minmax );
my ($min, $max) = minmax @numbers;

List::Util is part of core, but List::MoreUtils isn't.

--ikegami

[Sep 16, 2019] How can I capture multiple matches from the same Perl regex - Stack Overflow

Sep 16, 2019 | stackoverflow.com

How can I capture multiple matches from the same Perl regex? Ask Question Asked 9 years, 4 months ago Active 7 years, 4 months ago Viewed 35k times 24 1


brian d foy ,May 22, 2010 at 15:42

I'm trying to parse a single string and get multiple chunks of data out from the same string with the same regex conditions. I'm parsing a single HTML doc that is static (For an undisclosed reason, I can't use an HTML parser to do the job.) I have an expression that looks like:
$string =~ /\<img\ssrc\="(.*)"/;

and I want to get the value of $1. However, in the one string, there are many img tags like this, so I need something like an array returned (@1?) is this possible?

VolatileRig ,Jan 14, 2014 at 19:41

As Jim's answer, use the /g modifier (in list context or in a loop).

But beware of greediness, you dont want the .* to match more than necessary (and dont escape < = , they are not special).

while($string =~ /<img\s+src="(.*?)"/g ) {
  ...
}

Robert Wohlfarth ,May 21, 2010 at 18:44

@list = ($string =~ m/\<img\ssrc\="(.*)"/g);

The g modifier matches all occurences in the string. List context returns all of the matches. See the m// operator in perlop .

dalton ,May 21, 2010 at 18:42

You just need the global modifier /g at the end of the match. Then loop through until there are no matches remaining
my @matches;
while ($string =~ /\<img\ssrc\="(.*)"/g) {
        push(@matches, $1);
}

VolatileRig ,May 24, 2010 at 16:37

Use the /g modifier and list context on the left, as in
@result = $string =~ /\<img\ssrc\="(.*)"/g;

[Sep 16, 2019] Pretty-print for shell script

Sep 16, 2019 | stackoverflow.com

Benoit ,Oct 21, 2010 at 13:19

I'm looking for something similiar to indent but for (bash) scripts. Console only, no colorizing, etc.

Do you know of one ?

Jamie ,Sep 11, 2012 at 3:00

Vim can indent bash scripts. But not reformat them before indenting.
Backup your bash script, open it with vim, type gg=GZZ and indent will be corrected. (Note for the impatient: this overwrites the file, so be sure to do that backup!)

Though, some bugs with << (expecting EOF as first character on a line) e.g.

EDIT: ZZ not ZQ

Daniel Martí ,Apr 8, 2018 at 13:52

A bit late to the party, but it looks like shfmt could do the trick for you.

Brian Chrisman ,Sep 9 at 7:47

In bash I do this:
reindent() {
source <(echo "Zibri () {";cat "$1"; echo "}")
declare -f Zibri|head --lines=-1|tail --lines=+3 | sed -e "s/^\s\s\s\s//"
}

this eliminates comments and reindents the script "bash way".

If you have HEREDOCS in your script, they got ruined by the sed in the previous function.

So use:

reindent() {
source <(echo "Zibri () {";cat "$1"; echo "}")
declare -f Zibri|head --lines=-1|tail --lines=+3"
}

But all your script will have a 4 spaces indentation.

Or you can do:

reindent () 
{ 
    rstr=$(mktemp -u "XXXXXXXXXX");
    source <(echo "Zibri () {";cat "$1"|sed -e "s/^\s\s\s\s/$rstr/"; echo "}");
    echo '#!/bin/bash';
    declare -f Zibri | head --lines=-1 | tail --lines=+3 | sed -e "s/^\s\s\s\s//;s/$rstr/    /"
}

which takes care also of heredocs.

> ,

Found this http://www.linux-kheops.com/doc/perl/perl-aubert/fmt.script .

Very nice, only one thing i took out is the [...]->test substitution.

[Sep 16, 2019] A command-line HTML pretty-printer Making messy HTML readable - Stack Overflow

Notable quotes:
"... Have a look at the HTML Tidy Project: http://www.html-tidy.org/ ..."
Sep 16, 2019 | stackoverflow.com

nisetama ,Aug 12 at 10:33

I'm looking for recommendations for HTML pretty printers which fulfill the following requirements:

> ,

Have a look at the HTML Tidy Project: http://www.html-tidy.org/

The granddaddy of HTML tools, with support for modern standards.

There used to be a fork called tidy-html5 which since became the official thing. Here is its GitHub repository .

Tidy is a console application for Mac OS X, Linux, Windows, UNIX, and more. It corrects and cleans up HTML and XML documents by fixing markup errors and upgrading legacy code to modern standards.

For your needs, here is the command line to call Tidy:

[Sep 11, 2019] string - Extract substring in Bash - Stack Overflow

Sep 11, 2019 | stackoverflow.com

Jeff ,May 8 at 18:30

Given a filename in the form someletters_12345_moreleters.ext , I want to extract the 5 digits and put them into a variable.

So to emphasize the point, I have a filename with x number of characters then a five digit sequence surrounded by a single underscore on either side then another set of x number of characters. I want to take the 5 digit number and put that into a variable.

I am very interested in the number of different ways that this can be accomplished.

Berek Bryan ,Jan 24, 2017 at 9:30

Use cut :
echo 'someletters_12345_moreleters.ext' | cut -d'_' -f 2

More generic:

INPUT='someletters_12345_moreleters.ext'
SUBSTRING=$(echo $INPUT| cut -d'_' -f 2)
echo $SUBSTRING

JB. ,Jan 6, 2015 at 10:13

If x is constant, the following parameter expansion performs substring extraction:
b=${a:12:5}

where 12 is the offset (zero-based) and 5 is the length

If the underscores around the digits are the only ones in the input, you can strip off the prefix and suffix (respectively) in two steps:

tmp=${a#*_}   # remove prefix ending in "_"
b=${tmp%_*}   # remove suffix starting with "_"

If there are other underscores, it's probably feasible anyway, albeit more tricky. If anyone knows how to perform both expansions in a single expression, I'd like to know too.

Both solutions presented are pure bash, with no process spawning involved, hence very fast.

A Sahra ,Mar 16, 2017 at 6:27

Generic solution where the number can be anywhere in the filename, using the first of such sequences:
number=$(echo $filename | egrep -o '[[:digit:]]{5}' | head -n1)

Another solution to extract exactly a part of a variable:

number=${filename:offset:length}

If your filename always have the format stuff_digits_... you can use awk:

number=$(echo $filename | awk -F _ '{ print $2 }')

Yet another solution to remove everything except digits, use

number=$(echo $filename | tr -cd '[[:digit:]]')

sshow ,Jul 27, 2017 at 17:22

In case someone wants more rigorous information, you can also search it in man bash like this
$ man bash [press return key]
/substring  [press return key]
[press "n" key]
[press "n" key]
[press "n" key]
[press "n" key]

Result:

${parameter:offset}
       ${parameter:offset:length}
              Substring Expansion.  Expands to  up  to  length  characters  of
              parameter  starting  at  the  character specified by offset.  If
              length is omitted, expands to the substring of parameter  start‐
              ing at the character specified by offset.  length and offset are
              arithmetic expressions (see ARITHMETIC  EVALUATION  below).   If
              offset  evaluates  to a number less than zero, the value is used
              as an offset from the end of the value of parameter.  Arithmetic
              expressions  starting  with  a - must be separated by whitespace
              from the preceding : to be distinguished from  the  Use  Default
              Values  expansion.   If  length  evaluates to a number less than
              zero, and parameter is not @ and not an indexed  or  associative
              array,  it is interpreted as an offset from the end of the value
              of parameter rather than a number of characters, and the  expan‐
              sion is the characters between the two offsets.  If parameter is
              @, the result is length positional parameters beginning at  off‐
              set.   If parameter is an indexed array name subscripted by @ or
              *, the result is the length members of the array beginning  with
              ${parameter[offset]}.   A  negative  offset is taken relative to
              one greater than the maximum index of the specified array.  Sub‐
              string  expansion applied to an associative array produces unde‐
              fined results.  Note that a negative offset  must  be  separated
              from  the  colon  by  at least one space to avoid being confused
              with the :- expansion.  Substring indexing is zero-based  unless
              the  positional  parameters are used, in which case the indexing
              starts at 1 by default.  If offset  is  0,  and  the  positional
              parameters are used, $0 is prefixed to the list.

Aleksandr Levchuk ,Aug 29, 2011 at 5:51

Building on jor's answer (which doesn't work for me):
substring=$(expr "$filename" : '.*_\([^_]*\)_.*')

kayn ,Oct 5, 2015 at 8:48

I'm surprised this pure bash solution didn't come up:
a="someletters_12345_moreleters.ext"
IFS="_"
set $a
echo $2
# prints 12345

You probably want to reset IFS to what value it was before, or unset IFS afterwards!

zebediah49 ,Jun 4 at 17:31

Here's how i'd do it:
FN=someletters_12345_moreleters.ext
[[ ${FN} =~ _([[:digit:]]{5})_ ]] && NUM=${BASH_REMATCH[1]}

Note: the above is a regular expression and is restricted to your specific scenario of five digits surrounded by underscores. Change the regular expression if you need different matching.

TranslucentCloud ,Jun 16, 2014 at 13:27

Following the requirements

I have a filename with x number of characters then a five digit sequence surrounded by a single underscore on either side then another set of x number of characters. I want to take the 5 digit number and put that into a variable.

I found some grep ways that may be useful:

$ echo "someletters_12345_moreleters.ext" | grep -Eo "[[:digit:]]+" 
12345

or better

$ echo "someletters_12345_moreleters.ext" | grep -Eo "[[:digit:]]{5}" 
12345

And then with -Po syntax:

$ echo "someletters_12345_moreleters.ext" | grep -Po '(?<=_)\d+' 
12345

Or if you want to make it fit exactly 5 characters:

$ echo "someletters_12345_moreleters.ext" | grep -Po '(?<=_)\d{5}' 
12345

Finally, to make it be stored in a variable it is just need to use the var=$(command) syntax.

Darron ,Jan 9, 2009 at 16:13

Without any sub-processes you can:
shopt -s extglob
front=${input%%_+([a-zA-Z]).*}
digits=${front##+([a-zA-Z])_}

A very small variant of this will also work in ksh93.

user2350426

add a comment ,Aug 5, 2014 at 8:11
If we focus in the concept of:
"A run of (one or several) digits"

We could use several external tools to extract the numbers.
We could quite easily erase all other characters, either sed or tr:

name='someletters_12345_moreleters.ext'

echo $name | sed 's/[^0-9]*//g'    # 12345
echo $name | tr -c -d 0-9          # 12345

But if $name contains several runs of numbers, the above will fail:

If "name=someletters_12345_moreleters_323_end.ext", then:

echo $name | sed 's/[^0-9]*//g'    # 12345323
echo $name | tr -c -d 0-9          # 12345323

We need to use regular expresions (regex).
To select only the first run (12345 not 323) in sed and perl:

echo $name | sed 's/[^0-9]*\([0-9]\{1,\}\).*$/\1/'
perl -e 'my $name='$name';my ($num)=$name=~/(\d+)/;print "$num\n";'

But we could as well do it directly in bash (1) :

regex=[^0-9]*([0-9]{1,}).*$; \
[[ $name =~ $regex ]] && echo ${BASH_REMATCH[1]}

This allows us to extract the FIRST run of digits of any length
surrounded by any other text/characters.

Note : regex=[^0-9]*([0-9]{5,5}).*$; will match only exactly 5 digit runs. :-)

(1) : faster than calling an external tool for each short texts. Not faster than doing all processing inside sed or awk for large files.

codist ,May 6, 2011 at 12:50

Here's a prefix-suffix solution (similar to the solutions given by JB and Darron) that matches the first block of digits and does not depend on the surrounding underscores:
str='someletters_12345_morele34ters.ext'
s1="${str#"${str%%[[:digit:]]*}"}"   # strip off non-digit prefix from str
s2="${s1%%[^[:digit:]]*}"            # strip off non-digit suffix from s1
echo "$s2"                           # 12345

Campa ,Oct 21, 2016 at 8:12

I love sed 's capability to deal with regex groups:
> var="someletters_12345_moreletters.ext"
> digits=$( echo $var | sed "s/.*_\([0-9]\+\).*/\1/p" -n )
> echo $digits
12345

A slightly more general option would be not to assume that you have an underscore _ marking the start of your digits sequence, hence for instance stripping off all non-numbers you get before your sequence: s/[^0-9]\+\([0-9]\+\).*/\1/p .


> man sed | grep s/regexp/replacement -A 2
s/regexp/replacement/
    Attempt to match regexp against the pattern space.  If successful, replace that portion matched with replacement.  The replacement may contain the special  character  &  to
    refer to that portion of the pattern space which matched, and the special escapes \1 through \9 to refer to the corresponding matching sub-expressions in the regexp.

More on this, in case you're not too confident with regexps:

All escapes \ are there to make sed 's regexp processing work.

Dan Dascalescu ,May 8 at 18:28

Given test.txt is a file containing "ABCDEFGHIJKLMNOPQRSTUVWXYZ"
cut -b19-20 test.txt > test1.txt # This will extract chars 19 & 20 "ST" 
while read -r; do;
> x=$REPLY
> done < test1.txt
echo $x
ST

Alex Raj Kaliamoorthy ,Jul 29, 2016 at 7:41

My answer will have more control on what you want out of your string. Here is the code on how you can extract 12345 out of your string
str="someletters_12345_moreleters.ext"
str=${str#*_}
str=${str%_more*}
echo $str

This will be more efficient if you want to extract something that has any chars like abc or any special characters like _ or - . For example: If your string is like this and you want everything that is after someletters_ and before _moreleters.ext :

str="someletters_123-45-24a&13b-1_moreleters.ext"

With my code you can mention what exactly you want. Explanation:

#* It will remove the preceding string including the matching key. Here the key we mentioned is _ % It will remove the following string including the matching key. Here the key we mentioned is '_more*'

Do some experiments yourself and you would find this interesting.

Dan Dascalescu ,May 8 at 18:27

similar to substr('abcdefg', 2-1, 3) in php:
echo 'abcdefg'|tail -c +2|head -c 3

olibre ,Nov 25, 2015 at 14:50

Ok, here goes pure Parameter Substitution with an empty string. Caveat is that I have defined someletters and moreletters as only characters. If they are alphanumeric, this will not work as it is.
filename=someletters_12345_moreletters.ext
substring=${filename//@(+([a-z])_|_+([a-z]).*)}
echo $substring
12345

gniourf_gniourf ,Jun 4 at 17:33

There's also the bash builtin 'expr' command:
INPUT="someletters_12345_moreleters.ext"  
SUBSTRING=`expr match "$INPUT" '.*_\([[:digit:]]*\)_.*' `  
echo $SUBSTRING

russell ,Aug 1, 2013 at 8:12

A little late, but I just ran across this problem and found the following:
host:/tmp$ asd=someletters_12345_moreleters.ext 
host:/tmp$ echo `expr $asd : '.*_\(.*\)_'`
12345
host:/tmp$

I used it to get millisecond resolution on an embedded system that does not have %N for date:

set `grep "now at" /proc/timer_list`
nano=$3
fraction=`expr $nano : '.*\(...\)......'`
$debug nano is $nano, fraction is $fraction

> ,Aug 5, 2018 at 17:13

A bash solution:
IFS="_" read -r x digs x <<<'someletters_12345_moreleters.ext'

This will clobber a variable called x . The var x could be changed to the var _ .

input='someletters_12345_moreleters.ext'
IFS="_" read -r _ digs _ <<<"$input"

[Sep 10, 2019] How do I avoid an uninitialized value

Sep 10, 2019 | stackoverflow.com

marto ,Jul 15, 2011 at 16:52

I use this scrub function to clean up output from other functions.
#!/usr/bin/perl
use warnings;
use strict;
use Data::Dumper;

my %h = (
    a => 1,
    b => 1
    );

print scrub($h{c});

sub scrub {
    my $a = shift;

    return ($a eq '' or $a eq '~' or not defined $a) ? -1 : $a;
}

The problem occurs when I also would like to handle the case, where the key in a hash doesn't exist, which is shown in the example with scrub($h{c}) .

What change should be make to scrub so it can handle this case?

Sandra Schlichting ,Jun 22, 2017 at 19:00

You're checking whether $a eq '' before checking whether it's defined, hence the warning "Use of uninitialized value in string eq". Simply change the order of things in the conditional:
return (!defined($a) or $a eq '' or $a eq '~') ? -1 : $a;

As soon as anything in the chain of 'or's matches, Perl will stop processing the conditional, thus avoiding the erroneous attempt to compare undef to a string.

Sandra Schlichting ,Jul 14, 2011 at 14:34

In scrub it is too late to check, if the hash has an entry for key key . scrub() only sees a scalar, which is undef , if the hash key does not exist. But a hash could have an entry with the value undef also, like this:
my %h = (
 a => 1,
 b => 1,
 c => undef
);

So I suggest to check for hash entries with the exists function.

[Sep 10, 2019] How do I check if a Perl scalar variable has been initialized - Stack Overflow

Sep 10, 2019 | stackoverflow.com

How do I check if a Perl scalar variable has been initialized? Ask Question Asked 8 years, 11 months ago Active 3 years ago Viewed 49k times 33 10


brian d foy ,Sep 18, 2010 at 13:53

Is the following the best way to check if a scalar variable is initialized in Perl, using defined ?
my $var;

if (cond) {
    $var = "string1";
}

# Is this the correct way?
if (defined $var) {
    ...
}

mob ,Sep 25, 2010 at 21:35

Perl doesn't offer a way to check whether or not a variable has been initialized.

However, scalar variables that haven't been explicitly initialized with some value happen to have the value of undef by default. You are right about defined being the right way to check whether or not a variable has a value of undef .

There's several other ways tho. If you want to assign to the variable if it's undef , which your example code seems to indicate, you could, for example, use perl's defined-or operator:

$var //= 'a default value';

vol7ron ,Sep 17, 2010 at 23:17

It depends on what you're trying to do. The proper C way to do things is to initialize variables when they are declared; however, Perl is not C , so one of the following may be what you want:
  1)   $var = "foo" unless defined $var;      # set default after the fact
  2)   $var = defined $var? $var : {...};     # ternary operation
  3)   {...} if !(defined $var);              # another way to write 1)
  4)   $var = $var || "foo";                  # set to $var unless it's falsy, in which case set to 'foo'
  5)   $var ||= "foo";                        # retain value of $var unless it's falsy, in which case set to 'foo' (same as previous line)
  6)   $var = $var // "foo";                  # set to $var unless it's undefined, in which case set to 'foo'
  7)   $var //= "foo";                        # 5.10+ ; retain value of $var unless it's undefined, in which case set to 'foo' (same as previous line)


C way of doing things ( not recommended ):

# initialize the variable to a default value during declaration
#   then test against that value when you want to see if it's been changed
my $var = "foo";
{...}
if ($var eq "foo"){
   ... # do something
} else {
   ... # do something else
}

Another long-winded way of doing this is to create a class and a flag when the variable's been changed, which is unnecessary.

Axeman ,Sep 17, 2010 at 20:39

If you don't care whether or not it's empty, it is. Otherwise you can check
if ( length( $str || '' )) {}

swilliams ,Sep 17, 2010 at 20:53

It depends on what you plan on doing with the variable whether or not it is defined; as of Perl 5.10, you can do this (from perl51000delta ):

A new operator // (defined-or) has been implemented. The following expression:

 $a // $b

is merely equivalent to

defined $a ? $a : $b

and the statement

$c //= $d;

can now be used instead of

$c = $d unless defined $c;

rafl ,Jun 24, 2012 at 7:53

'defined' will return true if a variable has a real value.

As an aside, in a hash, this can be true:

if(exists $h{$e} && !defined $h{$e})

[Sep 10, 2019] logging - Perl - Output the log files - Stack Overflow

Aug 27, 2015 | stackoverflow.com

Perl - Output the log files Ask Question Asked 4 years ago Active 4 years ago Viewed 3k times 1 2


Arunesh Singh ,Aug 27, 2015 at 8:53

I have created a perl that telnet to multiple switches. I would like to check if telnet functions properly by telneting the switch.

This is my code to telnet to the switches:

#!/usr/bin/perl
use warnings;
use Net::Cisco;

open( OUTPUT, ">log.txt" );
open( SWITCHIP, "ip.txt" ) or die "couldn't open ip.txt";

my $count = 0;

while (<SWITCHIP>) {
    chomp($_);
    my $switch = $_;
    my $tl     = 0;
    my $t      = Net::Telnet::Cisco->new(
        Host => $switch,
        Prompt =>
            '/(?m:^(?:[\w.\/]+\:)?[\w.-]+\s?(?:\(config[^\)]*\))?\s?[\$#>]\s?(?:\(enable\))?\s*$)/',
        Timeout => 5,
        Errmode => 'return'
    ) or $tl = 1;

    my @output = ();
    if ( $tl != 1 ) {
        print "$switch Telnet success\n";
    }
    else {
        my $telnetstat = "Telnet Failed";
        print "$switch $telnetstat\n";
    }
    close(OUTPUT);
    $count++;
}

This is my output status after I was testing 7 switches:

10.xxx.3.17 Telnet success
10.xxx.10.12 Telnet success
10.xxx.136.10 Telnet success
10.xxx.136.12 Telnet success
10.xxx.188.188 Telnet Failed
10.xxx.136.13 Telnet success

I would like to convert the telnet result as log file.
How to separate successful and failed telnet results by using perl?

Danny Luk ,Aug 28, 2015 at 8:40

Please Try the following
#!/usr/bin/perl
use warnings;
use Net::Cisco;
################################### S
open( OUTPUTS, ">log_Success.txt" );
open( OUTPUTF, ">log_Fail.txt" );
################################### E
open( SWITCHIP, "ip.txt" ) or die "couldn't open ip.txt";

my $count = 0;

while (<SWITCHIP>) {
    chomp($_);
    my $switch = $_;
    my $tl     = 0;
    my $t      = Net::Telnet::Cisco->new(
        Host => $switch,
        Prompt =>
            '/(?m:^(?:[\w.\/]+\:)?[\w.-]+\s?(?:\(config[^\)]*\))?\s?[\$#>]\s?(?:\(enable\))?\s*$)/',
        Timeout => 5,
        Errmode => 'return'
    ) or $tl = 1;

    my @output = ();
################################### S
    if ( $tl != 1 ) {
        print "$switch Telnet success\n"; # for printing it in screen
        print OUTPUTS "$switch Telnet success\n"; # it will print it in the log_Success.txt
    }
    else {
        my $telnetstat = "Telnet Failed";
        print "$switch $telnetstat\n"; # for printing it in screen
        print OUTPUTF "$switch $telnetstat\n"; # it will print it in the log_Fail.txt
    }
################################### E
    $count++;
}
################################### S
close(SWITCHIP);
close(OUTPUTS);
close(OUTPUTF);
################################### E

Danny Luk ,Aug 28, 2015 at 8:39

In print statement after print just write the filehandle name which is OUTPUT in your code:
print OUTPUT "$switch Telnet success\n";

and

print OUTPUT "$switch $telnetstat\n";

A side note: always use a lexical filehandle and three arguments with error handling to open a file. This line open(OUTPUT, ">log.txt"); you can write like this:

open my $fhout, ">", "log.txt" or die $!;

Sobrique ,Aug 28, 2015 at 8:39

Use Sys::Syslog to write log messages.

But since you're opening a log.txt file with the handle OUTPUT , just change your two print statements to have OUTPUT as the first argument and the string as the next (without a comma).

my $telnetstat;
if($tl != 1) {
  $telnetstat = "Telnet success";
} else {
  $telnetstat = "Telnet Failed";
}
print OUTPUT "$switch $telnetstat\n";

# Or the shorter ternary operator line for all the above:
print OUTPUT $swtich . (!$tl ? " Telnet success\n" : " Telnet failed\n");

You might consider moving close to an END block:

END {
  close(OUTPUT);
}

Not only because it's in your while loop.

[Sep 08, 2019] How to replace spaces in file names using a bash script

Sep 08, 2019 | stackoverflow.com

Ask Question Asked 9 years, 4 months ago Active 2 months ago Viewed 226k times 238 127


Mark Byers ,Apr 25, 2010 at 19:20

Can anyone recommend a safe solution to recursively replace spaces with underscores in file and directory names starting from a given root directory? For example:
$ tree
.
|-- a dir
|   `-- file with spaces.txt
`-- b dir
    |-- another file with spaces.txt
    `-- yet another file with spaces.pdf

becomes:

$ tree
.
|-- a_dir
|   `-- file_with_spaces.txt
`-- b_dir
    |-- another_file_with_spaces.txt
    `-- yet_another_file_with_spaces.pdf

Jürgen Hötzel ,Nov 4, 2015 at 3:03

Use rename (aka prename ) which is a Perl script which may be on your system already. Do it in two steps:
find -name "* *" -type d | rename 's/ /_/g'    # do the directories first
find -name "* *" -type f | rename 's/ /_/g'

Based on Jürgen's answer and able to handle multiple layers of files and directories in a single bound using the "Revision 1.5 1998/12/18 16:16:31 rmb1" version of /usr/bin/rename (a Perl script):

find /tmp/ -depth -name "* *" -execdir rename 's/ /_/g' "{}" \;

oevna ,Jan 1, 2016 at 8:25

I use:
for f in *\ *; do mv "$f" "${f// /_}"; done

Though it's not recursive, it's quite fast and simple. I'm sure someone here could update it to be recursive.

The ${f// /_} part utilizes bash's parameter expansion mechanism to replace a pattern within a parameter with supplied string. The relevant syntax is ${parameter/pattern/string} . See: https://www.gnu.org/software/bash/manual/html_node/Shell-Parameter-Expansion.html or http://wiki.bash-hackers.org/syntax/pe .

armandino ,Dec 3, 2013 at 20:51

find . -depth -name '* *' \
| while IFS= read -r f ; do mv -i "$f" "$(dirname "$f")/$(basename "$f"|tr ' ' _)" ; done

failed to get it right at first, because I didn't think of directories.

Edmund Elmer ,Jul 3 at 7:12

you can use detox by Doug Harple
detox -r <folder>

Dennis Williamson ,Mar 22, 2012 at 20:33

A find/rename solution. rename is part of util-linux.

You need to descend depth first, because a whitespace filename can be part of a whitespace directory:

find /tmp/ -depth -name "* *" -execdir rename " " "_" "{}" ";"

armandino ,Apr 26, 2010 at 11:49

bash 4.0
#!/bin/bash
shopt -s globstar
for file in **/*\ *
do 
    mv "$file" "${file// /_}"       
done

Itamar ,Jan 31, 2013 at 21:27

you can use this:
    find . -name '* *' | while read fname 

do
        new_fname=`echo $fname | tr " " "_"`

        if [ -e $new_fname ]
        then
                echo "File $new_fname already exists. Not replacing $fname"
        else
                echo "Creating new file $new_fname to replace $fname"
                mv "$fname" $new_fname
        fi
done

yabt ,Apr 26, 2010 at 14:54

Here's a (quite verbose) find -exec solution which writes "file already exists" warnings to stderr:
function trspace() {
   declare dir name bname dname newname replace_char
   [ $# -lt 1 -o $# -gt 2 ] && { echo "usage: trspace dir char"; return 1; }
   dir="${1}"
   replace_char="${2:-_}"
   find "${dir}" -xdev -depth -name $'*[ \t\r\n\v\f]*' -exec bash -c '
      for ((i=1; i<=$#; i++)); do
         name="${@:i:1}"
         dname="${name%/*}"
         bname="${name##*/}"
         newname="${dname}/${bname//[[:space:]]/${0}}"
         if [[ -e "${newname}" ]]; then
            echo "Warning: file already exists: ${newname}" 1>&2
         else
            mv "${name}" "${newname}"
         fi
      done
  ' "${replace_char}" '{}' +
}

trspace rootdir _

degi ,Aug 8, 2011 at 9:10

This one does a little bit more. I use it to rename my downloaded torrents (no special characters (non-ASCII), spaces, multiple dots, etc.).
#!/usr/bin/perl

&rena(`find . -type d`);
&rena(`find . -type f`);

sub rena
{
    ($elems)=@_;
    @t=split /\n/,$elems;

    for $e (@t)
    {
    $_=$e;
    # remove ./ of find
    s/^\.\///;
    # non ascii transliterate
    tr [\200-\377][_];
    tr [\000-\40][_];
    # special characters we do not want in paths
    s/[ \-\,\;\?\+\'\"\!\[\]\(\)\@\#]/_/g;
    # multiple dots except for extension
    while (/\..*\./)
    {
        s/\./_/;
    }
    # only one _ consecutive
    s/_+/_/g;
    next if ($_ eq $e ) or ("./$_" eq $e);
    print "$e -> $_\n";
    rename ($e,$_);
    }
}

Junyeop Lee ,Apr 10, 2018 at 9:44

Recursive version of Naidim's Answers.
find . -name "* *" | awk '{ print length, $0 }' | sort -nr -s | cut -d" " -f2- | while read f; do base=$(basename "$f"); newbase="${base// /_}"; mv "$(dirname "$f")/$(basename "$f")" "$(dirname "$f")/$newbase"; done

ghoti ,Dec 5, 2016 at 21:16

I found around this script, it may be interesting :)
 IFS=$'\n';for f in `find .`; do file=$(echo $f | tr [:blank:] '_'); [ -e $f ] && [ ! -e $file ] && mv "$f" $file;done;unset IFS

ghoti ,Dec 5, 2016 at 21:17

Here's a reasonably sized bash script solution
#!/bin/bash
(
IFS=$'\n'
    for y in $(ls $1)
      do
         mv $1/`echo $y | sed 's/ /\\ /g'` $1/`echo "$y" | sed 's/ /_/g'`
      done
)

user1060059 ,Nov 22, 2011 at 15:15

This only finds files inside the current directory and renames them . I have this aliased.

find ./ -name "* *" -type f -d 1 | perl -ple '$file = $_; $file =~ s/\s+/_/g; rename($_, $file);

Hongtao ,Sep 26, 2014 at 19:30

I just make one for my own purpose. You may can use it as reference.
#!/bin/bash
cd /vzwhome/c0cheh1/dev_source/UB_14_8
for file in *
do
    echo $file
    cd "/vzwhome/c0cheh1/dev_source/UB_14_8/$file/Configuration/$file"
    echo "==> `pwd`"
    for subfile in *\ *; do [ -d "$subfile" ] && ( mv "$subfile" "$(echo $subfile | sed -e 's/ /_/g')" ); done
    ls
    cd /vzwhome/c0cheh1/dev_source/UB_14_8
done

Marcos Jean Sampaio ,Dec 5, 2016 at 20:56

For files in folder named /files
for i in `IFS="";find /files -name *\ *`
do
   echo $i
done > /tmp/list


while read line
do
   mv "$line" `echo $line | sed 's/ /_/g'`
done < /tmp/list

rm /tmp/list

Muhammad Annaqeeb ,Sep 4, 2017 at 11:03

For those struggling through this using macOS, first install all the tools:
 brew install tree findutils rename

Then when needed to rename, make an alias for GNU find (gfind) as find. Then run the code of @Michel Krelin:

alias find=gfind 
find . -depth -name '* *' \
| while IFS= read -r f ; do mv -i "$f" "$(dirname "$f")/$(basename "$f"|tr ' ' _)" ; done

[Sep 07, 2019] As soon as you stop writing code on a regular basis you stop being a programmer. You lose you qualification very quickly. That's a typical tragedy of talented programmers who became mediocre managers or, worse, theoretical computer scientists

Programming skills are somewhat similar to the skills of people who play violin or piano. As soon a you stop playing violin or piano still start to evaporate. First slowly, then quicker. In two yours you probably will lose 80%.
Notable quotes:
"... I happened to look the other day. I wrote 35 programs in January, and 28 or 29 programs in February. These are small programs, but I have a compulsion. I love to write programs and put things into it. ..."
Sep 07, 2019 | archive.computerhistory.org

Dijkstra said he was proud to be a programmer. Unfortunately he changed his attitude completely, and I think he wrote his last computer program in the 1980s. At this conference I went to in 1967 about simulation language, Chris Strachey was going around asking everybody at the conference what was the last computer program you wrote. This was 1967. Some of the people said, "I've never written a computer program." Others would say, "Oh yeah, here's what I did last week." I asked Edsger this question when I visited him in Texas in the 90s and he said, "Don, I write programs now with pencil and paper, and I execute them in my head." He finds that a good enough discipline.

I think he was mistaken on that. He taught me a lot of things, but I really think that if he had continued... One of Dijkstra's greatest strengths was that he felt a strong sense of aesthetics, and he didn't want to compromise his notions of beauty. They were so intense that when he visited me in the 1960s, I had just come to Stanford. I remember the conversation we had. It was in the first apartment, our little rented house, before we had electricity in the house.

We were sitting there in the dark, and he was telling me how he had just learned about the specifications of the IBM System/360, and it made him so ill that his heart was actually starting to flutter.

He intensely disliked things that he didn't consider clean to work with. So I can see that he would have distaste for the languages that he had to work with on real computers. My reaction to that was to design my own language, and then make Pascal so that it would work well for me in those days. But his response was to do everything only intellectually.

So, programming.

I happened to look the other day. I wrote 35 programs in January, and 28 or 29 programs in February. These are small programs, but I have a compulsion. I love to write programs and put things into it. I think of a question that I want to answer, or I have part of my book where I want to present something. But I can't just present it by reading about it in a book. As I code it, it all becomes clear in my head. It's just the discipline. The fact that I have to translate my knowledge of this method into something that the machine is going to understand just forces me to make that crystal-clear in my head. Then I can explain it to somebody else infinitely better. The exposition is always better if I've implemented it, even though it's going to take me more time.

[Sep 07, 2019] Knuth about computer science and money: At that point I made the decision in my life that I wasn't going to optimize my income;

Sep 07, 2019 | archive.computerhistory.org

So I had a programming hat when I was outside of Cal Tech, and at Cal Tech I am a mathematician taking my grad studies. A startup company, called Green Tree Corporation because green is the color of money, came to me and said, "Don, name your price. Write compilers for us and we will take care of finding computers for you to debug them on, and assistance for you to do your work. Name your price." I said, "Oh, okay. $100,000.", assuming that this was In that era this was not quite at Bill Gate's level today, but it was sort of out there.

The guy didn't blink. He said, "Okay." I didn't really blink either. I said, "Well, I'm not going to do it. I just thought this was an impossible number."

At that point I made the decision in my life that I wasn't going to optimize my income; I was really going to do what I thought I could do for well, I don't know. If you ask me what makes me most happy, number one would be somebody saying "I learned something from you". Number two would be somebody saying "I used your software". But number infinity would be Well, no. Number infinity minus one would be "I bought your book". It's not as good as "I read your book", you know. Then there is "I bought your software"; that was not in my own personal value. So that decision came up. I kept up with the literature about compilers. The Communications of the ACM was where the action was. I also worked with people on trying to debug the ALGOL language, which had problems with it. I published a few papers, like "The Remaining Trouble Spots in ALGOL 60" was one of the papers that I worked on. I chaired a committee called "Smallgol" which was to find a subset of ALGOL that would work on small computers. I was active in programming languages.

[Sep 07, 2019] Knuth: maybe 1 in 50 people have the "computer scientist's" type of intellect

Sep 07, 2019 | conservancy.umn.edu

Frana: You have made the comment several times that maybe 1 in 50 people have the "computer scientist's mind." Knuth: Yes. Frana: I am wondering if a large number of those people are trained professional librarians? [laughter] There is some strangeness there. But can you pinpoint what it is about the mind of the computer scientist that is....

Knuth: That is different?

Frana: What are the characteristics?

Knuth: Two things: one is the ability to deal with non-uniform structure, where you have case one, case two, case three, case four. Or that you have a model of something where the first component is integer, the next component is a Boolean, and the next component is a real number, or something like that, you know, non-uniform structure. To deal fluently with those kinds of entities, which is not typical in other branches of mathematics, is critical. And the other characteristic ability is to shift levels quickly, from looking at something in the large to looking at something in the small, and many levels in between, jumping from one level of abstraction to another. You know that, when you are adding one to some number, that you are actually getting closer to some overarching goal. These skills, being able to deal with nonuniform objects and to see through things from the top level to the bottom level, these are very essential to computer programming, it seems to me. But maybe I am fooling myself because I am too close to it.

Frana: It is the hardest thing to really understand that which you are existing within.

Knuth: Yes.

[Sep 07, 2019] Knuth: I can be a writer, who tries to organize other people's ideas into some kind of a more coherent structure so that it is easier to put things together

Sep 07, 2019 | conservancy.umn.edu

Knuth: I can be a writer, who tries to organize other people's ideas into some kind of a more coherent structure so that it is easier to put things together. I can see that I could be viewed as a scholar that does his best to check out sources of material, so that people get credit where it is due. And to check facts over, not just to look at the abstract of something, but to see what the methods were that did it and to fill in holes if necessary. I look at my role as being able to understand the motivations and terminology of one group of specialists and boil it down to a certain extent so that people in other parts of the field can use it. I try to listen to the theoreticians and select what they have done that is important to the programmer on the street; to remove technical jargon when possible.

But I have never been good at any kind of a role that would be making policy, or advising people on strategies, or what to do. I have always been best at refining things that are there and bringing order out of chaos. I sometimes raise new ideas that might stimulate people, but not really in a way that would be in any way controlling the flow. The only time I have ever advocated something strongly was with literate programming; but I do this always with the caveat that it works for me, not knowing if it would work for anybody else.

When I work with a system that I have created myself, I can always change it if I don't like it. But everybody who works with my system has to work with what I give them. So I am not able to judge my own stuff impartially. So anyway, I have always felt bad about if anyone says, 'Don, please forecast the future,'...

[Sep 06, 2019] Knuth: Programming and architecture are interrelated and it is impossible to create good architecure wthout actually programming at least of a prototype

Notable quotes:
"... When you're writing a document for a human being to understand, the human being will look at it and nod his head and say, "Yeah, this makes sense." But then there's all kinds of ambiguities and vagueness that you don't realize until you try to put it into a computer. Then all of a sudden, almost every five minutes as you're writing the code, a question comes up that wasn't addressed in the specification. "What if this combination occurs?" ..."
"... When you're faced with implementation, a person who has been delegated this job of working from a design would have to say, "Well hmm, I don't know what the designer meant by this." ..."
Sep 06, 2019 | archive.computerhistory.org

...I showed the second version of this design to two of my graduate students, and I said, "Okay, implement this, please, this summer. That's your summer job." I thought I had specified a language. I had to go away. I spent several weeks in China during the summer of 1977, and I had various other obligations. I assumed that when I got back from my summer trips, I would be able to play around with TeX and refine it a little bit. To my amazement, the students, who were outstanding students, had not competed [it]. They had a system that was able to do about three lines of TeX. I thought, "My goodness, what's going on? I thought these were good students." Well afterwards I changed my attitude to saying, "Boy, they accomplished a miracle."

Because going from my specification, which I thought was complete, they really had an impossible task, and they had succeeded wonderfully with it. These students, by the way, [were] Michael Plass, who has gone on to be the brains behind almost all of Xerox's Docutech software and all kind of things that are inside of typesetting devices now, and Frank Liang, one of the key people for Microsoft Word.

He did important mathematical things as well as his hyphenation methods which are quite used in all languages now. These guys were actually doing great work, but I was amazed that they couldn't do what I thought was just sort of a routine task. Then I became a programmer in earnest, where I had to do it. The reason is when you're doing programming, you have to explain something to a computer, which is dumb.

When you're writing a document for a human being to understand, the human being will look at it and nod his head and say, "Yeah, this makes sense." But then there's all kinds of ambiguities and vagueness that you don't realize until you try to put it into a computer. Then all of a sudden, almost every five minutes as you're writing the code, a question comes up that wasn't addressed in the specification. "What if this combination occurs?"

It just didn't occur to the person writing the design specification. When you're faced with implementation, a person who has been delegated this job of working from a design would have to say, "Well hmm, I don't know what the designer meant by this."

If I hadn't been in China they would've scheduled an appointment with me and stopped their programming for a day. Then they would come in at the designated hour and we would talk. They would take 15 minutes to present to me what the problem was, and then I would think about it for a while, and then I'd say, "Oh yeah, do this. " Then they would go home and they would write code for another five minutes and they'd have to schedule another appointment.

I'm probably exaggerating, but this is why I think Bob Floyd's Chiron compiler never got going. Bob worked many years on a beautiful idea for a programming language, where he designed a language called Chiron, but he never touched the programming himself. I think this was actually the reason that he had trouble with that project, because it's so hard to do the design unless you're faced with the low-level aspects of it, explaining it to a machine instead of to another person.

Forsythe, I think it was, who said, "People have said traditionally that you don't understand something until you've taught it in a class. The truth is you don't really understand something until you've taught it to a computer, until you've been able to program it." At this level, programming was absolutely important

[Sep 06, 2019] Knuth: No, I stopped going to conferences. It was too discouraging. Computer programming keeps getting harder because more stuff is discovered

Sep 06, 2019 | conservancy.umn.edu

Knuth: No, I stopped going to conferences. It was too discouraging. Computer programming keeps getting harder because more stuff is discovered. I can cope with learning about one new technique per day, but I can't take ten in a day all at once. So conferences are depressing; it means I have so much more work to do. If I hide myself from the truth I am much happier.

[Sep 06, 2019] How TAOCP was hatched

Notable quotes:
"... Also, Addison-Wesley was the people who were asking me to do this book; my favorite textbooks had been published by Addison Wesley. They had done the books that I loved the most as a student. For them to come to me and say, "Would you write a book for us?", and here I am just a secondyear gradate student -- this was a thrill. ..."
"... But in those days, The Art of Computer Programming was very important because I'm thinking of the aesthetical: the whole question of writing programs as something that has artistic aspects in all senses of the word. The one idea is "art" which means artificial, and the other "art" means fine art. All these are long stories, but I've got to cover it fairly quickly. ..."
Sep 06, 2019 | archive.computerhistory.org

Knuth: This is, of course, really the story of my life, because I hope to live long enough to finish it. But I may not, because it's turned out to be such a huge project. I got married in the summer of 1961, after my first year of graduate school. My wife finished college, and I could use the money I had made -- the $5000 on the compiler -- to finance a trip to Europe for our honeymoon.

We had four months of wedded bliss in Southern California, and then a man from Addison-Wesley came to visit me and said "Don, we would like you to write a book about how to write compilers."

The more I thought about it, I decided "Oh yes, I've got this book inside of me."

I sketched out that day -- I still have the sheet of tablet paper on which I wrote -- I sketched out 12 chapters that I thought ought to be in such a book. I told Jill, my wife, "I think I'm going to write a book."

As I say, we had four months of bliss, because the rest of our marriage has all been devoted to this book. Well, we still have had happiness. But really, I wake up every morning and I still haven't finished the book. So I try to -- I have to -- organize the rest of my life around this, as one main unifying theme. The book was supposed to be about how to write a compiler. They had heard about me from one of their editorial advisors, that I knew something about how to do this. The idea appealed to me for two main reasons. One is that I did enjoy writing. In high school I had been editor of the weekly paper. In college I was editor of the science magazine, and I worked on the campus paper as copy editor. And, as I told you, I wrote the manual for that compiler that we wrote. I enjoyed writing, number one.

Also, Addison-Wesley was the people who were asking me to do this book; my favorite textbooks had been published by Addison Wesley. They had done the books that I loved the most as a student. For them to come to me and say, "Would you write a book for us?", and here I am just a secondyear gradate student -- this was a thrill.

Another very important reason at the time was that I knew that there was a great need for a book about compilers, because there were a lot of people who even in 1962 -- this was January of 1962 -- were starting to rediscover the wheel. The knowledge was out there, but it hadn't been explained. The people who had discovered it, though, were scattered all over the world and they didn't know of each other's work either, very much. I had been following it. Everybody I could think of who could write a book about compilers, as far as I could see, they would only give a piece of the fabric. They would slant it to their own view of it. There might be four people who could write about it, but they would write four different books. I could present all four of their viewpoints in what I would think was a balanced way, without any axe to grind, without slanting it towards something that I thought would be misleading to the compiler writer for the future. I considered myself as a journalist, essentially. I could be the expositor, the tech writer, that could do the job that was needed in order to take the work of these brilliant people and make it accessible to the world. That was my motivation. Now, I didn't have much time to spend on it then, I just had this page of paper with 12 chapter headings on it. That's all I could do while I'm a consultant at Burroughs and doing my graduate work. I signed a contract, but they said "We know it'll take you a while." I didn't really begin to have much time to work on it until 1963, my third year of graduate school, as I'm already finishing up on my thesis. In the summer of '62, I guess I should mention, I wrote another compiler. This was for Univac; it was a FORTRAN compiler. I spent the summer, I sold my soul to the devil, I guess you say, for three months in the summer of 1962 to write a FORTRAN compiler. I believe that the salary for that was $15,000, which was much more than an assistant professor. I think assistant professors were getting eight or nine thousand in those days.

Feigenbaum: Well, when I started in 1960 at [University of California] Berkeley, I was getting $7,600 for the nine-month year.

Knuth: Knuth: Yeah, so you see it. I got $15,000 for a summer job in 1962 writing a FORTRAN compiler. One day during that summer I was writing the part of the compiler that looks up identifiers in a hash table. The method that we used is called linear probing. Basically you take the variable name that you want to look up, you scramble it, like you square it or something like this, and that gives you a number between one and, well in those days it would have been between 1 and 1000, and then you look there. If you find it, good; if you don't find it, go to the next place and keep on going until you either get to an empty place, or you find the number you're looking for. It's called linear probing. There was a rumor that one of Professor Feller's students at Princeton had tried to figure out how fast linear probing works and was unable to succeed. This was a new thing for me. It was a case where I was doing programming, but I also had a mathematical problem that would go into my other [job]. My winter job was being a math student, my summer job was writing compilers. There was no mix. These worlds did not intersect at all in my life at that point. So I spent one day during the summer while writing the compiler looking at the mathematics of how fast does linear probing work. I got lucky, and I solved the problem. I figured out some math, and I kept two or three sheets of paper with me and I typed it up. ["Notes on 'Open' Addressing', 7/22/63] I guess that's on the internet now, because this became really the genesis of my main research work, which developed not to be working on compilers, but to be working on what they call analysis of algorithms, which is, have a computer method and find out how good is it quantitatively. I can say, if I got so many things to look up in the table, how long is linear probing going to take. It dawned on me that this was just one of many algorithms that would be important, and each one would lead to a fascinating mathematical problem. This was easily a good lifetime source of rich problems to work on. Here I am then, in the middle of 1962, writing this FORTRAN compiler, and I had one day to do the research and mathematics that changed my life for my future research trends. But now I've gotten off the topic of what your original question was.

Feigenbaum: We were talking about sort of the.. You talked about the embryo of The Art of Computing. The compiler book morphed into The Art of Computer Programming, which became a seven-volume plan.

Knuth: Exactly. Anyway, I'm working on a compiler and I'm thinking about this. But now I'm starting, after I finish this summer job, then I began to do things that were going to be relating to the book. One of the things I knew I had to have in the book was an artificial machine, because I'm writing a compiler book but machines are changing faster than I can write books. I have to have a machine that I'm totally in control of. I invented this machine called MIX, which was typical of the computers of 1962.

In 1963 I wrote a simulator for MIX so that I could write sample programs for it, and I taught a class at Caltech on how to write programs in assembly language for this hypothetical computer. Then I started writing the parts that dealt with sorting problems and searching problems, like the linear probing idea. I began to write those parts, which are part of a compiler, of the book. I had several hundred pages of notes gathering for those chapters for The Art of Computer Programming. Before I graduated, I've already done quite a bit of writing on The Art of Computer Programming.

I met George Forsythe about this time. George was the man who inspired both of us [Knuth and Feigenbaum] to come to Stanford during the '60s. George came down to Southern California for a talk, and he said, "Come up to Stanford. How about joining our faculty?" I said "Oh no, I can't do that. I just got married, and I've got to finish this book first." I said, "I think I'll finish the book next year, and then I can come up [and] start thinking about the rest of my life, but I want to get my book done before my son is born." Well, John is now 40-some years old and I'm not done with the book. Part of my lack of expertise is any good estimation procedure as to how long projects are going to take. I way underestimated how much needed to be written about in this book. Anyway, I started writing the manuscript, and I went merrily along writing pages of things that I thought really needed to be said. Of course, it didn't take long before I had started to discover a few things of my own that weren't in any of the existing literature. I did have an axe to grind. The message that I was presenting was in fact not going to be unbiased at all. It was going to be based on my own particular slant on stuff, and that original reason for why I should write the book became impossible to sustain. But the fact that I had worked on linear probing and solved the problem gave me a new unifying theme for the book. I was going to base it around this idea of analyzing algorithms, and have some quantitative ideas about how good methods were. Not just that they worked, but that they worked well: this method worked 3 times better than this method, or 3.1 times better than this method. Also, at this time I was learning mathematical techniques that I had never been taught in school. I found they were out there, but they just hadn't been emphasized openly, about how to solve problems of this kind.

So my book would also present a different kind of mathematics than was common in the curriculum at the time, that was very relevant to analysis of algorithm. I went to the publishers, I went to Addison Wesley, and said "How about changing the title of the book from 'The Art of Computer Programming' to 'The Analysis of Algorithms'." They said that will never sell; their focus group couldn't buy that one. I'm glad they stuck to the original title, although I'm also glad to see that several books have now come out called "The Analysis of Algorithms", 20 years down the line.

But in those days, The Art of Computer Programming was very important because I'm thinking of the aesthetical: the whole question of writing programs as something that has artistic aspects in all senses of the word. The one idea is "art" which means artificial, and the other "art" means fine art. All these are long stories, but I've got to cover it fairly quickly.

I've got The Art of Computer Programming started out, and I'm working on my 12 chapters. I finish a rough draft of all 12 chapters by, I think it was like 1965. I've got 3,000 pages of notes, including a very good example of what you mentioned about seeing holes in the fabric. One of the most important chapters in the book is parsing: going from somebody's algebraic formula and figuring out the structure of the formula. Just the way I had done in seventh grade finding the structure of English sentences, I had to do this with mathematical sentences.

Chapter ten is all about parsing of context-free language, [which] is what we called it at the time. I covered what people had published about context-free languages and parsing. I got to the end of the chapter and I said, well, you can combine these ideas and these ideas, and all of a sudden you get a unifying thing which goes all the way to the limit. These other ideas had sort of gone partway there. They would say "Oh, if a grammar satisfies this condition, I can do it efficiently." "If a grammar satisfies this condition, I can do it efficiently." But now, all of a sudden, I saw there was a way to say I can find the most general condition that can be done efficiently without looking ahead to the end of the sentence. That you could make a decision on the fly, reading from left to right, about the structure of the thing. That was just a natural outgrowth of seeing the different pieces of the fabric that other people had put together, and writing it into a chapter for the first time. But I felt that this general concept, well, I didn't feel that I had surrounded the concept. I knew that I had it, and I could prove it, and I could check it, but I couldn't really intuit it all in my head. I knew it was right, but it was too hard for me, really, to explain it well.

So I didn't put in The Art of Computer Programming. I thought it was beyond the scope of my book. Textbooks don't have to cover everything when you get to the harder things; then you have to go to the literature. My idea at that time [is] I'm writing this book and I'm thinking it's going to be published very soon, so any little things I discover and put in the book I didn't bother to write a paper and publish in the journal because I figure it'll be in my book pretty soon anyway. Computer science is changing so fast, my book is bound to be obsolete.

It takes a year for it to go through editing, and people drawing the illustrations, and then they have to print it and bind it and so on. I have to be a little bit ahead of the state-of-the-art if my book isn't going to be obsolete when it comes out. So I kept most of the stuff to myself that I had, these little ideas I had been coming up with. But when I got to this idea of left-to-right parsing, I said "Well here's something I don't really understand very well. I'll publish this, let other people figure out what it is, and then they can tell me what I should have said." I published that paper I believe in 1965, at the end of finishing my draft of the chapter, which didn't get as far as that story, LR(k). Well now, textbooks of computer science start with LR(k) and take off from there. But I want to give you an idea of

[Sep 06, 2019] Python vs. Ruby Which is best for web development Opensource.com

Sep 06, 2019 | opensource.com

Python was developed organically in the scientific space as a prototyping language that easily could be translated into C++ if a prototype worked. This happened long before it was first used for web development. Ruby, on the other hand, became a major player specifically because of web development; the Rails framework extended Ruby's popularity with people developing complex websites.

Which programming language best suits your needs? Here is a quick overview of each language to help you choose:

Approach: one best way vs. human-language Python

Python takes a direct approach to programming. Its main goal is to make everything obvious to the programmer. In Python, there is only one "best" way to do something. This philosophy has led to a language strict in layout.

Python's core philosophy consists of three key hierarchical principles:

This regimented philosophy results in Python being eminently readable and easy to learn -- and why Python is great for beginning coders. Python has a big foothold in introductory programming courses . Its syntax is very simple, with little to remember. Because its code structure is explicit, the developer can easily tell where everything comes from, making it relatively easy to debug.

Python's hierarchy of principles is evident in many aspects of the language. Its use of whitespace to do flow control as a core part of the language syntax differs from most other languages, including Ruby. The way you indent code determines the meaning of its action. This use of whitespace is a prime example of Python's "explicit" philosophy, the shape a Python app takes spells out its logic and how the app will act.

Ruby

In contrast to Python, Ruby focuses on "human-language" programming, and its code reads like a verbal language rather than a machine-based one, which many programmers, both beginners and experts, like. Ruby follows the principle of " least astonishment ," and offers myriad ways to do the same thing. These similar methods can have multiple names, which many developers find confusing and frustrating.

Unlike Python, Ruby makes use of "blocks," a first-class object that is treated as a unit within a program. In fact, Ruby takes the concept of OOP (Object-Oriented Programming) to its limit. Everything is an object -- even global variables are actually represented within the ObjectSpace object. Classes and modules are themselves objects, and functions and operators are methods of objects. This ability makes Ruby especially powerful, especially when combined with its other primary strength: functional programming and the use of lambdas.

In addition to blocks and functional programming, Ruby provides programmers with many other features, including fragmentation, hashable and unhashable types, and mutable strings.

Ruby's fans find its elegance to be one of its top selling points. At the same time, Ruby's "magical" features and flexibility can make it very hard to track down bugs.

Communities: stability vs. innovation

Although features and coding philosophy are the primary drivers for choosing a given language, the strength of a developer community also plays an important role. Fortunately, both Python and Ruby boast strong communities.

Python

Python's community already includes a large Linux and academic community and therefore offers many academic use cases in both math and science. That support gives the community a stability and diversity that only grows as Python increasingly is used for web development.

Ruby

However, Ruby's community has focused primarily on web development from the get-go. It tends to innovate more quickly than the Python community, but this innovation also causes more things to break. In addition, while it has gotten more diverse, it has yet to reach the level of diversity that Python has.

Final thoughts

For web development, Ruby has Rails and Python has Django. Both are powerful frameworks, so when it comes to web development, you can't go wrong with either language. Your decision will ultimately come down to your level of experience and your philosophical preferences.

If you plan to focus on building web applications, Ruby is popular and flexible. There is a very strong community built upon it and they are always on the bleeding edge of development.

If you are interested in building web applications and would like to learn a language that's used more generally, try Python. You'll get a diverse community and lots of influence and support from the various industries in which it is used.

Tom Radcliffe - Tom Radcliffe has over 20 years experience in software development and management in both academia and industry. He is a professional engineer (PEO and APEGBC) and holds a PhD in physics from Queen's University at Kingston. Tom brings a passion for quantitative, data-driven processes to ActiveState .

[Sep 03, 2019] bash - How to convert strings like 19-FEB-12 to epoch date in UNIX - Stack Overflow

Feb 11, 2013 | stackoverflow.com

Asked 6 years, 6 months ago Active 2 years, 2 months ago Viewed 53k times 24 4

hellish ,Feb 11, 2013 at 3:45

In UNIX how to convert to epoch milliseconds date strings like:
19-FEB-12
16-FEB-12
05-AUG-09

I need this to compare these dates with the current time on the server.

> ,

To convert a date to seconds since the epoch:
date --date="19-FEB-12" +%s

Current epoch:

date +%s

So, since your dates are in the past:

NOW=`date +%s`
THEN=`date --date="19-FEB-12" +%s`

let DIFF=$NOW-$THEN
echo "The difference is: $DIFF"

Using BSD's date command, you would need

$ date -j -f "%d-%B-%y" 19-FEB-12 +%s

Differences from GNU date :

  1. -j prevents date from trying to set the clock
  2. The input format must be explicitly set with -f
  3. The input date is a regular argument, not an option (viz. -d )
  4. When no time is specified with the date, use the current time instead of midnight.

[Sep 03, 2019] command line - How do I convert an epoch timestamp to a human readable format on the cli - Unix Linux Stack Exchange

Sep 03, 2019 | unix.stackexchange.com

Gilles ,Oct 11, 2010 at 18:14

date -d @1190000000 Replace 1190000000 with your epoch

Stefan Lasiewski ,Oct 11, 2010 at 18:04

$ echo 1190000000 | perl -pe 's/(\d+)/localtime($1)/e' 
Sun Sep 16 20:33:20 2007

This can come in handy for those applications which use epoch time in the logfiles:

$ tail -f /var/log/nagios/nagios.log | perl -pe 's/(\d+)/localtime($1)/e'
[Thu May 13 10:15:46 2010] EXTERNAL COMMAND: PROCESS_SERVICE_CHECK_RESULT;HOSTA;check_raid;0;check_raid.pl: OK (Unit 0 on Controller 0 is OK)

Stéphane Chazelas ,Jul 31, 2015 at 20:24

With bash-4.2 or above:
printf '%(%F %T)T\n' 1234567890

(where %F %T is the strftime() -type format)

That syntax is inspired from ksh93 .

In ksh93 however, the argument is taken as a date expression where various and hardly documented formats are supported.

For a Unix epoch time, the syntax in ksh93 is:

printf '%(%F %T)T\n' '#1234567890'

ksh93 however seems to use its own algorithm for the timezone and can get it wrong. For instance, in Britain, it was summer time all year in 1970, but:

$ TZ=Europe/London bash -c 'printf "%(%c)T\n" 0'
Thu 01 Jan 1970 01:00:00 BST
$ TZ=Europe/London ksh93 -c 'printf "%(%c)T\n" "#0"'
Thu Jan  1 00:00:00 1970

DarkHeart ,Jul 28, 2014 at 3:56

Custom format with GNU date :
date -d @1234567890 +'%Y-%m-%d %H:%M:%S'

Or with GNU awk :

awk 'BEGIN { print strftime("%Y-%m-%d %H:%M:%S", 1234567890); }'

Linked SO question: https://stackoverflow.com/questions/3249827/convert-from-unixtime-at-command-line

,

The two I frequently use are:
$ perl -leprint\ scalar\ localtime\ 1234567890
Sat Feb 14 00:31:30 2009

[Sep 02, 2019] bash - Pretty-print for shell script

Oct 21, 2010 | stackoverflow.com

Pretty-print for shell script Ask Question Asked 8 years, 10 months ago Active 30 days ago Viewed 14k times


Benoit ,Oct 21, 2010 at 13:19

I'm looking for something similiar to indent but for (bash) scripts. Console only, no colorizing, etc.

Do you know of one ?

Jamie ,Sep 11, 2012 at 3:00

Vim can indent bash scripts. But not reformat them before indenting.
Backup your bash script, open it with vim, type gg=GZZ and indent will be corrected. (Note for the impatient: this overwrites the file, so be sure to do that backup!)

Though, some bugs with << (expecting EOF as first character on a line) e.g.

EDIT: ZZ not ZQ

Daniel Martí ,Apr 8, 2018 at 13:52

A bit late to the party, but it looks like shfmt could do the trick for you.

Brian Chrisman ,Aug 11 at 4:08

In bash I do this:
reindent() {
source <(echo "Zibri () {";cat "$1"; echo "}")
declare -f Zibri|head --lines=-1|tail --lines=+3 | sed -e "s/^\s\s\s\s//"
}

this eliminates comments and reindents the script "bash way".

If you have HEREDOCS in your script, they got ruined by the sed in the previous function.

So use:

reindent() {
source <(echo "Zibri () {";cat "$1"; echo "}")
declare -f Zibri|head --lines=-1|tail --lines=+3"
}

But all your script will have a 4 spaces indentation.

Or you can do:

reindent () 
{ 
    rstr=$(cat /dev/urandom | tr -dc 'a-zA-Z0-9' | fold -w 16 | head -n 1);
    source <(echo "Zibri () {";cat "$1"|sed -e "s/^\s\s\s\s/$rstr/"; echo "}");
    echo '#!/bin/bash';
    declare -f Zibri | head --lines=-1 | tail --lines=+3 | sed -e "s/^\s\s\s\s//;s/$rstr/    /"
}

which takes care also of heredocs.

Pius Raeder ,Jan 10, 2017 at 8:35

Found this http://www.linux-kheops.com/doc/perl/perl-aubert/fmt.script .

Very nice, only one thing i took out is the [...]->test substitution.

[Sep 02, 2019] How to get the current line number of a file open using Perl

Sep 02, 2019 | stackoverflow.com

How to get the current line number of a file open using Perl? Ask Question Asked 8 years, 3 months ago Active 6 months ago Viewed 33k times 25 1


tadmc ,May 8, 2011 at 17:08

open my $fp, '<', $file or die $!;

while (<$fp>) {
    my $line = $_;
    if ($line =~ /$regex/) {
        # How do I find out which line number this match happened at?
    }
}

close $fp;

tchrist ,Apr 22, 2015 at 21:16

Use $. (see perldoc perlvar ).

tchrist ,May 7, 2011 at 16:48

You can also do it through OO interface:
use IO::Handle;
# later on ...
my $n = $fp->input_line_number();

This is in perldoc perlvar , too.

> ,

Don't use $. , nor $_ or any global variable. Use this instead:
while(my $line = <FILE>) {
  print $line unless ${\*FILE}->input_line_number == 1;
}

To avoid this and a lot of others Perl gotchas you can use on Atom or VSCode packages like linter-perl . Stop making Perl a write-only language !

[Aug 31, 2019] Complexity prevent programmer from ever learning the whole language, only subset is learned and used

Aug 31, 2019 | ask.slashdot.org

Re:Neither! (Score 2, Interesting) 817 b