Win a copy of Terraform in Action this week in the Cloud forum!
  • Post Reply Bookmark Topic Watch Topic
  • New Topic
programming forums Java Mobile Certification Databases Caching Books Engineering Micro Controllers OS Languages Paradigms IDEs Build Tools Frameworks Application Servers Open Source This Site Careers Other Pie Elite all forums
this forum made possible by our volunteer staff, including ...
Marshals:
  • Tim Cooke
  • Campbell Ritchie
  • Paul Clapham
  • Ron McLeod
  • Liutauras Vilda
Sheriffs:
  • Jeanne Boyarsky
  • Rob Spoor
  • Bear Bibeault
Saloon Keepers:
  • Jesse Silverman
  • Tim Moores
  • Stephan van Hulst
  • Tim Holloway
  • Carey Brown
Bartenders:
  • Piet Souris
  • Al Hobbs
  • salvin francis

Why are we allowed to call only UDFs and not functions from pyspark dataframe operations

 
Ranch Hand
Posts: 2550
13
  • Mark post as helpful
  • send pies
    Number of slices to send:
    Optional 'thank-you' note:
  • Quote
  • Report post to moderator
Suppose we have to populate value of a new column myCol based on some computation value returned from some function say myFunc

Instead of calling myFunc like in below code which is not allowed


We will have to create a UDF for myFunc and then call it.That will work.

What is the reason spark doesn't allow us to call function like this but allows only UDFs at this place ?


Thanks
 
reply
    Bookmark Topic Watch Topic
  • New Topic