• Post Reply Bookmark Topic Watch Topic
  • New Topic
programming forums Java Mobile Certification Databases Caching Books Engineering Micro Controllers OS Languages Paradigms IDEs Build Tools Frameworks Application Servers Open Source This Site Careers Other Pie Elite all forums
this forum made possible by our volunteer staff, including ...
Marshals:
  • Campbell Ritchie
  • Paul Clapham
  • Ron McLeod
  • Tim Cooke
  • Junilu Lacar
Sheriffs:
  • Rob Spoor
  • Devaka Cooray
  • Jeanne Boyarsky
Saloon Keepers:
  • Jesse Silverman
  • Stephan van Hulst
  • Tim Moores
  • Carey Brown
  • Tim Holloway
Bartenders:
  • Jj Roberts
  • Al Hobbs
  • Piet Souris

Python learnings

 
Saloon Keeper
Posts: 1609
52
Eclipse IDE Postgres Database C++ Java
  • Mark post as helpful
  • send pies
    Number of slices to send:
    Optional 'thank-you' note:
  • Quote
  • Report post to moderator
Normally in both Java and Python we don't need to even think about garbage collection, except when we do because there are some kinds of tiher memory or performance issues plaguing us.

A big difference between Java and Python is that Python uses for primarily historical reasons, both reference counting and the kind of garbage collection that we usually see in Java (there are various options available in Java for those who need more control for performance reasons or are just obsessive).

In my mind systems either used reference counting or the advanced stuff that detects islands of isolation and circular references, not both.  Well, Python does both.

It is not among the first, second or third wave of stuff you should know about when learning Python, but I kept wondering what it does and why I saw some behaviors I saw, so this article gives a good overview:
https://rushter.com/blog/python-garbage-collector/
 
Jesse Silverman
Saloon Keeper
Posts: 1609
52
Eclipse IDE Postgres Database C++ Java
  • Mark post as helpful
  • send pies
    Number of slices to send:
    Optional 'thank-you' note:
  • Quote
  • Report post to moderator
[cross-post warning] I posted this as a reply to a comment where someone was confused about why the definition of the sum of two Circle objects would be a Circle object with a radius equal to the sum of the two Circle objects being added.  That so many different presentations fail to suggest people use some common sense when overloading, rather than just doing it because you can and it is cool, suggests to me that my normal interpretation of Java's decision to omit the ability to overload operators to protect everyone from unreadable, unanalyzable code as Overly Conservative may not have been fair.  While Python's own standard library seems to only overload things with their obvious and most clear meaning, even discussing the operators they decided to overload and some words as to why, which is great -- most of the people teaching introductory Python thru videos give some pretty ridiculous and fairly questionable examples while showing the mechanics, and do not warn beginners that they should take care to overload operators only in ways that are highly useful and make obvious sense.

When I took actual courses in C++, and in some of the better books, they emphasize that operator overloading is a possibility in C++, but it is prone to abuse, and can make your code harder to understand rather than easier, if you are not careful.  It should be used when there is an obvious, clear interpretation to what the use of the operator would mean, maybe so much so that people would be disappointed that it didn't work that way if you didn't do it.  None of the presentations of operator overloading in Python that I have watched yet seems to emphasize this "With great power comes great responsibility" aspect.  One of the anti-patterns or examples of how not to do things is usually a non-intuitive definition of +.  What indeed does it mean to add two Dog objects, two Network connections, two Circle objects?  If people are going to disagree on what the obvious interpretation is, it is a sign that maybe it isn't helpful to be overloading that operator.  Of course, the "magic" or Dunder Method model that we use to implement the overloading has other, more obviously generally useful applications, such as __eq__(), __str__() and the most famous, __init__() so the videos are still useful.  I wish they paid just a little bit of attention to suggest when it is actually good to overload a particular operator for a particular class or maybe not so good an idea.
 
Jesse Silverman
Saloon Keeper
Posts: 1609
52
Eclipse IDE Postgres Database C++ Java
  • Mark post as helpful
  • send pies
    Number of slices to send:
    Optional 'thank-you' note:
  • Quote
  • Report post to moderator
One more thing.
There are tons of differences in how OOPS concepts are implemented, but one of the many weird ones is that it has class/static variable overriding with inheritance, rather than hiding.

It is almost easier to do OOPS in Java if you've never done Python and vice versa.

Otherwise, it is appropriately named, because you will be saying "OOPS!!" a lot as your misconceptions about how OOPS works fail to carry across the language barrier.
 
Jesse Silverman
Saloon Keeper
Posts: 1609
52
Eclipse IDE Postgres Database C++ Java
  • Mark post as helpful
  • send pies
    Number of slices to send:
    Optional 'thank-you' note:
  • Quote
  • Report post to moderator
So I'm doing a lot of new stuff in Python, well, it is new to me.

I had a nightmare last night where I was writing code to deal with stuff and couldn't tell if the code was correct or not.

We have had discussions about to what extent it is true that you can be writing totally invalid Python code that couldn't possibly work and not knowing until you executed the exact right lines with exact right data.

Right at the beginning of loading your whole source (a bit weird talking about Python source when normally there are no compiled or linked artifacts on disk, except for .pyc cached byte-code files) it does indeed check for a bunch of gross syntactical violations in modern CPython 3.9, all sorts of stuff like unmatched parenthesis or [ ] or { } or " " or ' '.  So what some people were saying about earlier versions of Python that you could have absolute line noise or a sneeze and not know until runtime wasn't quite so true anymore.

However, a disturbingly large category of errors that wouldn't even compile in traditional statically-typed compiled languages can make it all the way thru to testing, and a missing test case can mean that they make it right on in to production.

Static typing and compiled languages don't reduce this to zero.  There are many types of errors that can easily be made in C code, for instance, unwittingly returning a pointer to an about-to-vanish stack variable, attempting to free a piece of static memory, freeing a pointer that you weren't supposed to free there because someone else was still using it to free later, etc. that can cause disaster far away in a large program, with attendant debugging nightmares, despite 'strict error checking at compile time'.  Java explicitly set out to make as many of such things literally impossible as problems anymore, mostly but not entirely by garbage collecting everything and having no explicit freeing of memory anywhere.  It succeeded admirably for many classes of errors, but there are still places where type erasure with generics or failure to properly override .equals() or use of mutable data types in situations where immutability is required for correctness can bite you, and most of these are covered on the OCJP.  Without proper knowledge and attention to these things, it is still possible to make Java code that happily compiles and runs but has invalid stuff in it that isn't just business logic (using the wrong formula to calculate interest or the area of a shape) but actually is using the language in an unsafe way that is destined to cause runtime exceptions, or worse yet, wrong results that look plausible and will only be caught by sufficient testing.  This is much more noticeable in multi-threaded code, where the dangers of writing something that compiles but will have, perhaps sporadic deadlocks or even flat-out wrong results is much higher.  They have attempted to address this by giving us newer models for multi-threading access, and encouraging more leveraging of immutability and functional styles to eliminate hazards of shared mutable objects being accessed by multiple threads.

Static analyzers for C, C++, C# and Java, which are facilitated by their static nature, can catch incredibly subtle bugs in all of these languages.  They are often expensive commercial products for commercial code, but worth it, and are generally free for open source code use -- that works two ways, they generate good will in the open source community and they can develop and improve their analysis code with lots and lots of free real-world test cases.  I am a huge fan of employing these for complex code that needs to work correctly.

Back to my nightmare.  While you won't often have code running that is grossly syntactically malformed, large classes of logic errors can creep in that might not be seen until runtime with just the right combinations of data to trigger them.  There are many things that could happen in Python code, where a user decides to directly mess with something that isn't tightly encapsulated that will cause crashes or huge bugs in behavior of even the best-tested classes.  You can say "don't do that" and with some work can basically emulate strong encapsulation using Python language conventions involving names with _ or __ prefixes.

Also, while Python has remained dynamic, it has added opt-in static declarations where anything looking at them can tell whether they contain internal inconsistencies or other code calling them is right or not.  It is harder, I believe, to engineer reliable 'linters' (an old term from the earlier days of static analysis that we'd all but stopped using in terms of native code because we'd gone so far beyond that, but whatever) but Python is popular and used for many important things and people are working hard at them.  They will often give us the familiar, comforting squiggles in our IDE's on code that would have 'compiled' and executed just fine until it didn't, allowing us to never check those bugs in in the first place.

Anyway, in my nightmare, I was on some tight deadline for a project, trying to get some tricky new code just right, and had the sick feeling that I didn't know "whether Python even understood" what I was typing.  I was going to find out when I did careful and detailed testing, which due to time pressure I questioned whether I was going to even be able to complete properly.  The levels of mistakes I was suspecting I was making would have definitely been failed compiles in static languages I have worked successfully in, and the lack of PyLint or PyLance squiggles in my IDE gave me no additional confidence that my code would be correctly interpreted as I wrote it.  This was a nightmare, after all, and I was dreaming, but the concern wasn't imaginary.  The obvious solution is to have plenty of tests, and to ensure that having all the proper and appropriate tests is part of the definition of done and scheduled in time estimates for completion of any new code or updates or re-factorings.  Those are all things that too commonly get shortchanged in too many environments.

Anyway, this is not one way.  There are myriad disasters one can easily cause with inattentive, rushed or sloppy native code in C or C++ that are banished forever in both Java and Python.  Java can catch countless usage errors that even modern linters would probably miss on Python, and Python won't run your code if you try to place mutable objects in a Set or use them as keys, to give an example of where Python is 'safer' than Java in the hands of the less expert, or more tired or rushed programmer.

Most of the things that seem insane that Python allows you to do can be easily avoided with some discipline.  If you don't like the types of variables changing as you move thru your code, well, don't write:


If you think it is horrible that ten different instances of your same class can all have different attributes from each other, rather than just different values  for the same attributes of the same type, well, never do THAT.

I remember when I was very first learning C as a kid, the grad students who worked as TA's would say "Ugh--these students think their code is good because it compiles!!" and the first few times I heard it while trying to fix compile errors I was like "better than mine which still doesn't??" but of course they were referring to all the many, many kinds of usage errors that would only be found during proper testing, many of which are no longer possible in Java and Python, others of which still are.

Also talking about various languages people would say "Just because a language has some feature doesn't mean you should use it!" -- which sounded weird to someone who was used to C or Pascal but now makes eminent sense.  Whichever language you are using, if there are things that make you less comfortable about the correctness, readability, or reliability of your code, just say "No, thanks."

I was playing with tkinter, noticing that the anchor value seemed not to be a str type but an Enum, as it wasn't in quotes, and wondered how it got that modernized because tkinter long predated Enums in Python.

Of course, it wasn't any Enum type, but a simple constant giving a name for a string literal.  I immediately tried writing NE, SW = SW, NE changing the values of the compass constants and re-running my code.  The alignment of everything changed!  Well, don't do that!

That happened to be some of the last code I'd written before I went to bed...

I guess nightmares can be avoided in any language with sufficient understanding, judgement and discipline, and encountered without.
 
Saloon Keeper
Posts: 24501
167
Android Eclipse IDE Tomcat Server Redhat Java Linux
  • Mark post as helpful
  • send pies
    Number of slices to send:
    Optional 'thank-you' note:
  • Quote
  • Report post to moderator
Here's a classic Python "gotcha".

I'm developing a home automation/environmental monitoring system based on the Domoticz platform. I've found that in many cases the easiest way to talk to it is via its web API and the  easiest way to talk to the web API has been Python.

I decided that I wanted to add temperature and humidity sensing to the device that measures barometric pressure.. In the process I re-designed the interface code. The primary data is a text line consisting of name/value pairs like so:

p=102395,pi=30.32,alt=-89.69,t0=28.62,t1=28.30,h=45.20

So I wrote a loop to split on the commas and an inner loop to enter the name/value into a dictionary (splitting at the "=").

So far so good, except that the barometric pressure was too high and needed to be scaled. I did a quick edit and restarted the processor. No data came in.

Read the logs and changing made it blow up. Because pressure (the value assigned to "p") was a string, not a number. I had to change it to

Now as it happens, the downstream code didn't have to change, since I used "pressure" as a format argument and the format function didn't care if it was string or float or int or boolean. But a strong-typed compiler would have flagged the defective code before I tried to run it. Or commit it to the production server.

Incidentally, "alt" is supposed to be altitude and I have no idea why the sensor thinks I'm 90m below sea level. But I don't use that info anyway.
 
Ranch Hand
Posts: 2543
13
  • Mark post as helpful
  • send pies
    Number of slices to send:
    Optional 'thank-you' note:
  • Quote
  • Report post to moderator

Tim Holloway wrote:. But a strong-typed compiler would have flagged the defective code before I tried to run it. Or commit it to the production server.


One disadvantage of not being a compiled language.

But would IDE like IntelliJ not show this in red? I see it showing in red for syntax problems.
 
Tim Holloway
Saloon Keeper
Posts: 24501
167
Android Eclipse IDE Tomcat Server Redhat Java Linux
  • Mark post as helpful
  • send pies
    Number of slices to send:
    Optional 'thank-you' note:
  • Quote
  • Report post to moderator

Monica Shiralkar wrote:But would IDE like IntelliJ not show this in red? I see it showing in red for syntax problems.



Possibly, but I edited this file on a server using the Nano text editor. My servers do not have GUIs - a GUI demands resources I'd rather give to actual applications.
 
Jesse Silverman
Saloon Keeper
Posts: 1609
52
Eclipse IDE Postgres Database C++ Java
  • Mark post as helpful
  • send pies
    Number of slices to send:
    Optional 'thank-you' note:
  • Quote
  • Report post to moderator
My friend mentioned this and I wanted to slug him, but it is true:
All the Linters that run so nicely and in-the-background in your IDE can also be run by hand from the shell command line as well.

So weird how far our definitions of "annoying" have been changed by kushy, comfy IDE's.
 
Jesse Silverman
Saloon Keeper
Posts: 1609
52
Eclipse IDE Postgres Database C++ Java
  • Mark post as helpful
  • send pies
    Number of slices to send:
    Optional 'thank-you' note:
  • Quote
  • Report post to moderator
Tim, if your operation was % or * you would have gotten a runtime string formatting error because your format tuple wasn't the right type, or 1.021.021.021.021.02 perhaps!
Those have legal meanings in Python (at least syntactically) but probably not what you intended if you thought you had a float and an int at that moment!

Not sure how much the combo of opt-in type declarations and linters would have helped because I haven't started using them, but maybe a lot?
 
Tim Holloway
Saloon Keeper
Posts: 24501
167
Android Eclipse IDE Tomcat Server Redhat Java Linux
  • Mark post as helpful
  • send pies
    Number of slices to send:
    Optional 'thank-you' note:
  • Quote
  • Report post to moderator

Jesse Silverman wrote:Tim, if your operation was % or * you would have gotten a runtime string formatting error because your format tuple wasn't the right type, or 1.021.021.021.021.02 perhaps!



I'm not sure, but that sounds more like Python 2.

Anyway, I prefer actual testing to "lint" programs. This isn't something whose coding needs to meet a shop standard (which is the other thing that lint programs often check). And yes,

I was doing this on a hot production server, where I've learned not to install build tools, since that's what gave me my one and only worm, circa 1999.
 
Jesse Silverman
Saloon Keeper
Posts: 1609
52
Eclipse IDE Postgres Database C++ Java
  • Mark post as helpful
  • send pies
    Number of slices to send:
    Optional 'thank-you' note:
  • Quote
  • Report post to moderator
 
Tim Holloway
Saloon Keeper
Posts: 24501
167
Android Eclipse IDE Tomcat Server Redhat Java Linux
  • Mark post as helpful
  • send pies
    Number of slices to send:
    Optional 'thank-you' note:
  • Quote
  • Report post to moderator
Disclaimer: the machine in question has no direct connection with the Internet. It's more like a very intelligent router between the DMZ and back office and contains the cache server for OS updates and some internal-only web services. And, incidentally, the primary testing was done elsewhere, but this was a 1-line fix and that machine has special hardware on it.

So no one's credit card or other sensitive information was even indirectly at risk here.

My critical stuff is in Java, and I'm not as casual about that!
 
Jesse Silverman
Saloon Keeper
Posts: 1609
52
Eclipse IDE Postgres Database C++ Java
  • Mark post as helpful
  • send pies
    Number of slices to send:
    Optional 'thank-you' note:
  • Quote
  • Report post to moderator
I thought as much, I was joking about what some others may do that sounds superficially similar but definitely is not.
 
Jesse Silverman
Saloon Keeper
Posts: 1609
52
Eclipse IDE Postgres Database C++ Java
  • Mark post as helpful
  • send pies
    Number of slices to send:
    Optional 'thank-you' note:
  • Quote
  • Report post to moderator
The April Fool's moose is awesome, I needed an excuse to post something just so I could say that.

I noticed today that Python has finally got its version of final/Final, which is to say just a loose general guideline of ideal suggested behavior:
https://stackoverflow.com/questions/802578/final-keyword-equivalent-for-variables-in-python

And also that my hazy memory that it had come to C++ as well was also correct:
https://www.geeksforgeeks.org/c-final-specifier/

and that moose, tho!

Technically, it isn't a keyword in either Python or C++, but it is there and it came there from Java...
 
Tim Holloway
Saloon Keeper
Posts: 24501
167
Android Eclipse IDE Tomcat Server Redhat Java Linux
  • Mark post as helpful
  • send pies
    Number of slices to send:
    Optional 'thank-you' note:
  • Quote
  • Report post to moderator
That's kind of funny, since as I recall, the original C++ could only overload if the base method was defined with the "overload" attribute. Now it appears we've gone the other way around.
 
Jesse Silverman
Saloon Keeper
Posts: 1609
52
Eclipse IDE Postgres Database C++ Java
  • Mark post as helpful
  • send pies
    Number of slices to send:
    Optional 'thank-you' note:
  • Quote
  • Report post to moderator
Do we mean overload, which required the innovation of 'name mangling' (gosh that always did and still sounds horrible, like when someone pronounces the new programmer's name all wrong because they aren't familiar with Indian or Nordic names) and work on the linker, versus C which did and still has no overloads?  This still today doesn't exist in standard Python, the last definition of a given function/method name wins,  and is based on static compile-time type of the reference in Java....

Or do we mean override, (which I have always had co-workers which confused the word with overwrite), which needs an entry in the VTable and is automatic in C#, Python and Java, but required the virtual keyword for a method to get added to the VTable in C++?

I don't think I am being pedantic when I insist that the interpretation of OOPS in all these languages has as many differences as similarities, despite people ignoring them and just talking about a mythical (to me) "Generic OOPS Competence".  The number of places where reasoning about Class and Object behavior from one language to another can bite you in the moose butt are....large!

Throw in all the variations from different generation of language standard versions (how many places can you still find that "Java does not allow method implementations s in interfaces" for instance?) -- and you have more interpretations of "Basic OOPS Principles" than I can count.

I was reading up on "Why is Python so slow?" (depends on which flavor, they all have different speeds for different things and different levels of compatibility with modern CPython 3.9.x in different areas) and discussions on Guido's blog about various attempts to add opt-in static typing and realized what I had guessed before but that nothing I read had said explicitly, you can not just dynamically add data methods to individual Python object instances, but methods/behavior as well.  When they say 'dynamic', holy crap they really mean it!  I am still not sure whether methods can be added or removed from individual instances or the whole class at once or both, and you can always say, like with changing the values of constants, "if you don't like it, don't do it" -- a message that appeals strongly to my Libertarian leanings in the general social contract but freaks me out on something used for production code...both the strongly typed languages and the dynamic languages have various features that can be considered 'safer' and 'less safe' and man it is a lot to take in and remember if you are regularly working across languages.
 
Monica Shiralkar
Ranch Hand
Posts: 2543
13
  • Mark post as helpful
  • send pies
    Number of slices to send:
    Optional 'thank-you' note:
  • Quote
  • Report post to moderator
I have seen that when people from Java background start working on Python they mistakenly keep saying main method whereas in Python it is not main method but main function.
 
Jesse Silverman
Saloon Keeper
Posts: 1609
52
Eclipse IDE Postgres Database C++ Java
  • Mark post as helpful
  • send pies
    Number of slices to send:
    Optional 'thank-you' note:
  • Quote
  • Report post to moderator

Monica Shiralkar wrote:I have seen that when people from Java background start working on Python they mistakenly keep saying main method whereas in Python it is not main method but main function.



While this is true, I mean True, it is the least of what's going on in terms of Java mindset being confusing to Python mindset people and vice versa.

Even in C++, only Java people call the member functions methods for the most part.  The stand-alone functions that aren't members of a class, even Java people hopefully don't call methods in Python and C++?

Most of the things I have posted on this thread represent deep, conceptual differences in thinking about things that have similar names.

In Python, five members of the same class could potentially all have different data members -- not different values, but different *members*.
That can never happen in Java/C++/C#, the name of a class tells you all the data members any instance of it will ever have....

And lots of other things that have same or similar names but pretty different ideas about how the Universe works.

But yeah, main function defined in startup module vs. main method of top-level class or interface, for sure.
 
Tim Holloway
Saloon Keeper
Posts: 24501
167
Android Eclipse IDE Tomcat Server Redhat Java Linux
  • Mark post as helpful
  • send pies
    Number of slices to send:
    Optional 'thank-you' note:
  • Quote
  • Report post to moderator

Jesse Silverman wrote:
In Python, five members of the same class could potentially all have different data members -- not different values, but different *members*.
That can never happen in Java/C++/C#, the name of a class tells you all the data members any instance of it will ever have....



To clarify, five instances of the same class could potentially all have different (named) data members.

This is also true of JavaScript, I believe. object-oriented implementations of LISP, and possibly Smalltalk.
 
Jesse Silverman
Saloon Keeper
Posts: 1609
52
Eclipse IDE Postgres Database C++ Java
  • Mark post as helpful
  • send pies
    Number of slices to send:
    Optional 'thank-you' note:
  • Quote
  • Report post to moderator
Thanks Tim, definitely meant instances, of course.

They will say that Smalltalk influenced Java, but not in that way for sure!
 
Jesse Silverman
Saloon Keeper
Posts: 1609
52
Eclipse IDE Postgres Database C++ Java
  • Mark post as helpful
  • send pies
    Number of slices to send:
    Optional 'thank-you' note:
  • Quote
  • Report post to moderator
I've started doing a lot more NumPy.

While for people who've been doing maths programming forever won't necessarily find a lot new here, it is the one Pythonic way to do what had been achieved with many slightly, annoyingly different ways to get the same effects before.

I mostly really like it, and definitely appreciate the fact that nearly everything maths-based sits on top of it these days.

Like UML, it certainly isn't perfect, but just having SO many different ways of doing the same things just drives me nuts.

There may be one or two things that people miss about some of the things it supplanted.

Then again, the widespread adoption suggests to me that maybe not, or if there was, they've gotten over it.

From Matlab to PowerBI, there's a lot of ways to do math stuff.

But NumPy leaves you sitting in the driver's seat of "One of the world's easiest, and one of the most popular" programming languages, instead of out on some island.

Go NumPy!!
 
Jesse Silverman
Saloon Keeper
Posts: 1609
52
Eclipse IDE Postgres Database C++ Java
  • Mark post as helpful
  • send pies
    Number of slices to send:
    Optional 'thank-you' note:
  • Quote
  • Report post to moderator
For reference, the two most annoying things about Python remain its lack of commitment to the Advanced Computer Science notion of constants, and several things that can all be summed up as 'excessive commitment to dynamism'.

Python is so dynamic it reminds me of someone coming to work tripping balls on mushrooms, not just micro-dosing.

Sure, the last ten thousand times thru the loop the loan payment didn't have a favorite color or any pizza toppings, but hey, maybe this time thru it does, better check just in case.

It is "interesting" just how dynamic everything is, but maybe I am old-fashioned, I like relatively uptight and static definitions of things.

I am also a bit iffy about some aspects of scoping, but suspect there are optional features I can enlist that will make those vanish.

Long term, I think the various places that optional static typing can be introduced would induce me to do so if I was opening up code as an API or something.
The flexibilities there seem awesome as long as you are coding for yourself, but a little too fail-at-runtime for things being consumed by complete strangers...

I guess the Pythonic way of thinking about it is that we are all adults and we will be careful...I still like seat belts and air bags despite the fact they occasionally kill people and many find them annoying...


 
Don't get me started about those stupid light bulbs.
reply
    Bookmark Topic Watch Topic
  • New Topic