Win a copy of Zero to AI - A non-technical, hype-free guide to prospering in the AI era this week in the Artificial Intelligence and Machine Learning forum!
  • Post Reply Bookmark Topic Watch Topic
  • New Topic
programming forums Java Mobile Certification Databases Caching Books Engineering Micro Controllers OS Languages Paradigms IDEs Build Tools Frameworks Application Servers Open Source This Site Careers Other all forums
this forum made possible by our volunteer staff, including ...
Marshals:
  • Campbell Ritchie
  • Liutauras Vilda
  • Paul Clapham
  • Bear Bibeault
  • Jeanne Boyarsky
Sheriffs:
  • Ron McLeod
  • Tim Cooke
  • Devaka Cooray
Saloon Keepers:
  • Tim Moores
  • Tim Holloway
  • Jj Roberts
  • Stephan van Hulst
  • Carey Brown
Bartenders:
  • salvin francis
  • Scott Selikoff
  • fred rosenberger

Can we test our Spark code on local eclipse which should run fine on cluster ?

 
Ranch Foreman
Posts: 1770
9
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
Spark code is developed first on local using typically Eclipse and run by setting master as local. Later the jar is run on cluster of multiple nodes.

However when we are running on local using eclipse, we are dealing with single node whereas there will be multiple nodes on cluster. Given this ,whatever code we test by running on local on Eclipse , will it always run fine on cluster with multiple nodes too ?  Which means that can the code be tested on local or will there be some cases where no matter if we have tested on local on Eclipse ,it may still not work as it is on cluster because it has multiple nodes ? Thanks
 
Don't get me started about those stupid light bulbs.
reply
    Bookmark Topic Watch Topic
  • New Topic