HDC调试需求开发（15万预算）,能者速来！>>> 2015-08-26 16:48:55,941 INFO [main] spark.SecurityManager (Logging.scala:logInfo(59)) - Changing view acls to: root2015-08-26 16:48:55,946 INFO [main] spark.SecurityManager (Logging.scala:logInfo(59)) - Changing modify acls to: root 2015-08-26 16:48:55,947 INFO [main] spark.SecurityManager (Logging.scala:logInfo(59)) - SecurityManager: authentication disabled; ui acls disabled; users with view permissions: Set(root); users with modify permissions: Set(root) 2015-08-26 16:48:56,403 INFO [sparkDriver-akka.actor.default-dispatcher-4] slf4j.Slf4jLogger (Slf4jLogger.scala:applyOrElse(80)) - Slf4jLogger started 2015-08-26 16:48:56,456 INFO [sparkDriver-akka.actor.default-dispatcher-4] Remoting (Slf4jLogger.scala:apply$mcV$sp(74)) - Starting remoting 2015-08-26 16:48:56,816 INFO [sparkDriver-akka.actor.default-dispatcher-5] Remoting (Slf4jLogger.scala:apply$mcV$sp(74)) - Remoting started; listening on addresses :[akka.tcp://sparkDriver@hadoop05:54033] 2015-08-26 16:48:56,820 INFO [sparkDriver-akka.actor.default-dispatcher-5] Remoting (Slf4jLogger.scala:apply$mcV$sp(74)) - Remoting now listens on addresses: [akka.tcp://sparkDriver@hadoop05:54033] 2015-08-26 16:48:56,832 INFO [main] util.Utils (Logging.scala:logInfo(59)) - Successfully started service 'sparkDriver' on port 54033. 2015-08-26 16:48:56,857 INFO [main] spark.SparkEnv (Logging.scala:logInfo(59)) - Registering MapOutputTracker 2015-08-26 16:48:56,870 INFO [main] spark.SparkEnv (Logging.scala:logInfo(59)) - Registering BlockManagerMaster 2015-08-26 16:48:56,889 INFO [main] storage.DiskBlockManager (Logging.scala:logInfo(59)) - Created local directory at /tmp/spark-local-20150826164856-d539 2015-08-26 16:48:56,893 INFO [main] storage.MemoryStore (Logging.scala:logInfo(59)) - MemoryStore started with capacity 530.3 MB 2015-08-26 16:48:57,347 WARN [main] util.NativeCodeLoader (NativeCodeLoader.java:<clinit>(62)) - Unable to load native-hadoop library for your platform... using builtin-java classes where applicable 2015-08-26 16:48:57,538 INFO [main] spark.HttpFileServer (Logging.scala:logInfo(59)) - HTTP File server directory is /tmp/spark-8ea80eb5-5ae9-4f11-8c5f-4616d14fc3d3 2015-08-26 16:48:57,543 INFO [main] spark.HttpServer (Logging.scala:logInfo(59)) - Starting HTTP Server 2015-08-26 16:48:57,603 INFO [main] server.Server (Server.java:doStart(272)) - jetty-8.y.z-SNAPSHOT 2015-08-26 16:48:57,624 INFO [main] server.AbstractConnector (AbstractConnector.java:doStart(338)) - Started SocketConnector@0.0.0.0:50271 2015-08-26 16:48:57,624 INFO [main] util.Utils (Logging.scala:logInfo(59)) - Successfully started service 'HTTP file server' on port 50271. 2015-08-26 16:48:57,739 INFO [main] server.Server (Server.java:doStart(272)) - jetty-8.y.z-SNAPSHOT 2015-08-26 16:48:57,752 INFO [main] server.AbstractConnector (AbstractConnector.java:doStart(338)) - Started SelectChannelConnector@0.0.0.0:4040 2015-08-26 16:48:57,753 INFO [main] util.Utils (Logging.scala:logInfo(59)) - Successfully started service 'SparkUI' on port 4040. 2015-08-26 16:48:57,756 INFO [main] ui.SparkUI (Logging.scala:logInfo(59)) - Started SparkUI at http://hadoop05:4040 2015-08-26 16:48:57,885 INFO [main] spark.SparkContext (Logging.scala:logInfo(59)) - Added JAR /home/hbase/cdrp-0.0.2-SNAPSHOT.jar at http://10.2.98.144:50271/jars/cdrp-0.0.2-SNAPSHOT.jar with timestamp 1440578937884 2015-08-26 16:48:57,886 INFO [main] spark.SparkContext (Logging.scala:logInfo(59)) - Added JAR /home/hbase/oozie-client-4.0.0-cdh5.2.0.jar at http://10.2.98.144:50271/jars/oozie-client-4.0.0-cdh5.2.0.jar with timestamp 1440578937886 2015-08-26 16:48:58,052 INFO [main] client.RMProxy (RMProxy.java:createRMProxy(98)) - Connecting to ResourceManager at /0.0.0.0:8032 2015-08-26 16:48:58,283 INFO [main] yarn.Client (Logging.scala:logInfo(59)) - Requesting a new application from cluster with 7 NodeManagers 2015-08-26 16:48:58,298 INFO [main] yarn.Client (Logging.scala:logInfo(59)) - Verifying our application has not requested more than the maximum memory capability of the cluster (9990 MB per container) 2015-08-26 16:48:58,299 INFO [main] yarn.Client (Logging.scala:logInfo(59)) - Will allocate AM container, with 1408 MB memory including 384 MB overhead 2015-08-26 16:48:58,300 INFO [main] yarn.Client (Logging.scala:logInfo(59)) - Setting up container launch context for our AM 2015-08-26 16:48:58,305 INFO [main] yarn.Client (Logging.scala:logInfo(59)) - Preparing resources for our AM container 2015-08-26 16:48:58,442 INFO [main] yarn.Client (Logging.scala:logInfo(59)) - Source and destination file systems are the same. Not copying file:/opt/cloudera/parcels/CDH-5.3.5-1.cdh5.3.5.p0.4/jars/spark-assembly-1.2.0-cdh5.3.5-hadoop2.5.0-cdh5.3.5.jar 2015-08-26 16:48:58,526 INFO [main] yarn.Client (Logging.scala:logInfo(59)) - Setting up the launch environment for our AM container 2015-08-26 16:48:58,731 INFO [main] spark.SecurityManager (Logging.scala:logInfo(59)) - Changing view acls to: root 2015-08-26 16:48:58,732 INFO [main] spark.SecurityManager (Logging.scala:logInfo(59)) - Changing modify acls to: root 2015-08-26 16:48:58,732 INFO [main] spark.SecurityManager (Logging.scala:logInfo(59)) - SecurityManager: authentication disabled; ui acls disabled; users with view permissions: Set(root); users with modify permissions: Set(root) 2015-08-26 16:48:58,735 INFO [main] yarn.Client (Logging.scala:logInfo(59)) - Submitting application 42 to ResourceManager 2015-08-26 16:48:58,776 INFO [main] impl.YarnClientImpl (YarnClientImpl.java:submitApplication(236)) - Submitted application application_1440484199657_0042 2015-08-26 16:48:59,782 INFO [main] yarn.Client (Logging.scala:logInfo(59)) - Application report for application_1440484199657_0042 (state: ACCEPTED) 2015-08-26 16:48:59,790 INFO [main] yarn.Client (Logging.scala:logInfo(59)) - client token: N/A diagnostics: N/A ApplicationMaster host: N/A ApplicationMaster RPC port: -1 queue: root.root start time: 1440578938748 final status: UNDEFINED tracking URL: http://hadoop05:8088/proxy/application_1440484199657_0042/ user: root 2015-08-26 16:49:00,794 INFO [main] yarn.Client (Logging.scala:logInfo(59)) - Application report for application_1440484199657_0042 (state: ACCEPTED) 2015-08-26 16:49:01,798 INFO [main] yarn.Client (Logging.scala:logInfo(59)) - Application report for application_1440484199657_0042 (state: ACCEPTED) 2015-08-26 16:49:02,803 INFO [main] yarn.Client (Logging.scala:logInfo(59)) - Application report for application_1440484199657_0042 (state: ACCEPTED) 2015-08-26 16:49:03,807 INFO [main] yarn.Client (Logging.scala:logInfo(59)) - Application report for application_1440484199657_0042 (state: ACCEPTED) 2015-08-26 16:49:04,812 INFO [main] yarn.Client (Logging.scala:logInfo(59)) - Application report for application_1440484199657_0042 (state: ACCEPTED) 2015-08-26 16:49:05,818 INFO [main] yarn.Client (Logging.scala:logInfo(59)) - Application report for application_1440484199657_0042 (state: ACCEPTED) 2015-08-26 16:49:06,823 INFO [main] yarn.Client (Logging.scala:logInfo(59)) - Application report for application_1440484199657_0042 (state: ACCEPTED) 2015-08-26 16:49:07,828 INFO [main] yarn.Client (Logging.scala:logInfo(59)) - Application report for application_1440484199657_0042 (state: ACCEPTED) 2015-08-26 16:49:08,832 INFO [main] yarn.Client (Logging.scala:logInfo(59)) - Application report for application_1440484199657_0042 (state: ACCEPTED) 2015-08-26 16:49:09,837 INFO [main] yarn.Client (Logging.scala:logInfo(59)) - Application report for application_1440484199657_0042 (state: ACCEPTED) 2015-08-26 16:49:10,844 INFO [main] yarn.Client (Logging.scala:logInfo(59)) - Application report for application_1440484199657_0042 (state: ACCEPTED) 2015-08-26 16:49:11,849 INFO [main] yarn.Client (Logging.scala:logInfo(59)) - Application report for application_1440484199657_0042 (state: ACCEPTED) 2015-08-26 16:49:12,853 INFO [main] yarn.Client (Logging.scala:logInfo(59)) - Application report for application_1440484199657_0042 (state: FAILED) 2015-08-26 16:49:12,855 INFO [main] yarn.Client (Logging.scala:logInfo(59)) - client token: N/A diagnostics: Application application_1440484199657_0042 failed 2 times due to AM Container for appattempt_1440484199657_0042_000002 exited with exitCode: 1 due to: Exception from container-launch. Container id: container_1440484199657_0042_02_000001 Exit code: 1 Stack trace: ExitCodeException exitCode=1: at org.apache.hadoop.util.Shell.runCommand(Shell.java:538) at org.apache.hadoop.util.Shell.run(Shell.java:455) at org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:702) at org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor.launchContainer(DefaultContainerExecutor.java:197) at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:299) at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:81) at java.util.concurrent.FutureTask.run(FutureTask.java:262) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread.java:745) Container exited with a non-zero exit code 1 .Failing this attempt.. Failing the application. ApplicationMaster host: N/A ApplicationMaster RPC port: -1 queue: root.root start time: 1440578938748 final status: FAILED tracking URL: http://hadoop05:8088/cluster/app/application_1440484199657_0042 user: root Exception in thread "main" org.apache.spark.SparkException: Yarn application has already ended! It might have been killed or unable to launch application master. at org.apache.spark.scheduler.cluster.YarnClientSchedulerBackend.waitForApplication(YarnClientSchedulerBackend.scala:102) at org.apache.spark.scheduler.cluster.YarnClientSchedulerBackend.start(YarnClientSchedulerBackend.scala:58) at org.apache.spark.scheduler.TaskSchedulerImpl.start(TaskSchedulerImpl.scala:140) at org.apache.spark.SparkContext.<init>(SparkContext.scala:335) at org.apache.spark.api.java.JavaSparkContext.<init>(JavaSparkContext.scala:61) at com.goodwill.cdr.hadoop.spark.PatientCountJHMK.execute(PatientCountJHMK.java:56) at com.goodwill.cdr.hadoop.spark.PatientCountJHMK.main(PatientCountJHMK.java:40) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:606) at org.apache.spark.deploy.SparkSubmit$.launch(SparkSubmit.scala:358) at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:75) at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)